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ABSTRACT 


A  study  was  conducted  to  examine  various  aspects  of  Distance  Learning  (DL) 
applications  currently  under  review  by  the  Marine  Corps,  and  determine  whether  these 
programs,  if  initiated,  provide  a  positive  Return  on  Investment  (ROI).  The  objective  was 
to  determine  how  DL  applications  may  be  applied  in  the  most  advantageous  manner,  to 
increase  the  overall  efficiency  of  current  training  programs  from  both  a  monetary  and 
quality  perspective.  Specifically,  DL  applications  were  evaluated  for  pertinence  to  the 
four  categories  of  learners  found  within  the  organizational  hierarchy.  To  accomplish  this 
objective,  information  was  collected  from  the  DL  Branch,  Training  &  Education 
Division,  HQMC,  as  well  as  from  faculty  and  staff  at  the  Marine  Corps  Communications 
and  Electronics  Course,  Marine  Corps  Air  Ground  Combat  Center,  29  Palms,  CA. 
Results  were  favorable  with  a  positive  ROI  being  determined  from  the  stated 
assumptions.  Other  findings  included  that  the  most  beneficial  application  of  DL 
technology  should  be  primarily  toward  advanced  level  training  with  possible 
considerations  for  Marines  awaiting  training,  and  that  due  to  increased  instructional 
requirements,  the  timesavings  attributed  to  advances  in  training  technology  should  not 
automatically  result  in  reductions  in  formal  course  curricula.  Simply  stated,  DL 
technologies  provide  great  value  added  potential  to  enhance  knowledge  transfer  in 
today’s  dynamic  and  fluid  training  environment,  but  should  be  viewed  primarily  as  a 
complement  to,  rather  than  replacement  for,  traditional  instructional  methods. 
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I.  INTRODUCTION 


A.  OVERVIEW 

1.  Background 

In  the  winter  of  1944  the  U.S.  Army  in  Europe  faced  a  critical  manpower 
shortage,  particularly  in  regard  to  combat  infantrymen.  These  shortages  stemmed  from 
the  tremendous  losses  in  personnel  taken  during  the  proceeding  months  of  campaigning 
in  France  and  Belgium.  In  an  effort  to  alleviate  these  shortages,  the  allied  high  command 
directed  that  the  flow  of  replacements  from  bases  in  England  and  the  United  States  be 
substantially  increased.  To  meet  this  requirement,  the  quality  and  length  of  training  new 
soldiers  received  prior  to  arrival  at  the  front  had  to  be  reduced  to  get  them  there  at  a  faster 
rate.  Replacements  were  thus  sent  into  combat  inadequately  trained  and  as  a  consequence 
suffered  inordinately  high  casualty  rates.  Ambrose  (1997)  discusses  how  the  casualty 
rates  of  typical  replacements  exceeded  50  percent,  and  normally  occurred  in  less  than  72 
hours  after  their  arrival  simply  because  they  lacked  basic  survival  skills.  Such 
information,  he  adds,  was  relatively  easy  to  pass  along  and  could  be  taught  in  a  very  short 
period  of  time,  if  only  such  time  had  been  made  available.  Ambrose  (1997)  blames  the 
entire  American  chain  of  command  for  this  problem.  He  states  that  the  plan  was 
unquestioningly  endorsed  by  the  Army  Chief  of  Staff,  General  Marshall,  and  the  Supreme 
Allied  Commander,  General  Eisenhower,  to  muffle  complaints  from  field  commanders, 
such  as  Generals  Bradley  and  Patton,  who  consistently  clamored  for  more  fresh  troops. 
All  were  found  to  share  equal  responsibility  for  casualty  rates  that  are  now  viewed  as 
excessive  and  wasteful  (Ambrose,  1997). 

In  pursuit  of  this  objective,  it  would  later  be  found  that  many  proposals  for 
increased  training  efficiency  actually  caused  more  casualties  than  they  helped  prevent,  as 
critical  information  was  eliminated  in  the  interest  of  increased  throughput.  Interestingly 
enough,  similar  problems  would  arise  again  in  both  Korea  and  Vietnam  as  the  U.S.  Army 
shortened  training  pipelines  to  get  troops  into  combat  at  a  faster  rate  (Ambrose,  1997). 
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2.  The  Problem 


What  relevance  do  such  anecdotes  have  for  today’s  military?  In  the  current 
environment  of  highly  complex,  technically  advanced  combat  information  systems,  it  is 
imperative  that  personnel  are  as  proficient  as  possible  at  their  jobs.  To  gain  and  maintain 
such  proficiency  is  becoming  exceedingly  difficult,  however,  as  the  quantity  of  pertinent 
information  is  growing  at  a  rate  far  in  excess  of  the  time  available  to  master  it.  The 
problem  is  thus  how  to  maximize  the  potential  of  training  and  education  methods  while 
minimizing  the  time  required  for  actual  knowledge  transfer. 

3.  A  Proposed  Solution 

Current  training  technologies  offer  a  solution  to  this  problem  by  affording  the 
student  more  opportunity  to  tailor  learning  to  their  individual  lifestyles  without 
sacrificing  the  quality  of  instruction.  This  paper  looks  at  one  aspect  of  these  advanced 
training  technologies  in  the  form  of  distance  learning.  Distance  learning  technology  has 
already  been  used  with  great  success  in  civilian  industry,  academia,  and  by  government 
agencies,  to  include  the  other  services.  The  Marine  Corps  quickly  recognized  the 
advantages  of  such  technology  and  recently  invested  quite  heavily  in  the  supporting 
infrastructure.  This  paper  will  attempt  to  measure  the  relative  return  that  may  be  realized 
from  such  investment,  particularly  in  regard  to  the  more  intangible  aspects  of  quality 
instruction  and  its  contribution  to  unit  readiness. 

4.  Consequences  of  Inaction 

Failure  to  properly  address  the  problem  of  providing  quality  instruction  in  a 
decreased  time  horizon  poses  severe  consequences  for  the  military.  Overall  unit 
readiness  will  obviously  suffer  with  spillover  effects  that  include  morale  problems  due  to 
a  lack  of  self-assurance,  which  may  further  lead  to  potential  retention  and  recruitment 
difficulties.  Most  significantly,  however,  the  military  may  be  doomed  to  repeat  the 
mistakes  of  the  past,  as  inadequately  trained  personnel  are  sent  in  harm’s  way  and  pay  the 
ultimate  price  for  their  ignorance. 
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B.  PURPOSE 


The  purpose  of  this  research  is  to  examine  various  aspects  of  Distance  Learning 
(DL)  applications  currently  under  review  by  the  USMC  and  determine  whether  these 
programs,  if  initiated,  provide  a  positive  Return  on  Investment  (ROI).  The  original  goal 
was  to  evaluate  DL  as  it  relates  to  the  following  categories  of  learners  found  within  the 
organizational  hierarchy:  (1)  Initial  skills  training;  (2)  Advanced  skills  training;  (3) 
Professional  Military  Education  (PME);  and  (4)  Marines  awaiting  training 
(MAT)/General  Interest.  It  was  quickly  discovered,  however,  that  to  fully  evaluate  all  the 
above  categories  in  the  limited  time  available  was  clearly  beyond  the  scope  of  this 
research,  so  the  focus  was  reduced  to  categories  one,  two,  and  four  with  an  emphasis  on 
categories  one  and  two.  Following  this  evaluation,  the  feasibility  of  implementing  DL  to 
improve  the  efficiency  of  current  training  programs  is  reviewed  through  empirical  study 
and  economic  analyses. 


C.  THE  NATURE  OF  THE  PROBLEM 


1.  The  Opportunity  Cost  of  Training. 

Where  possible,  the  Marine  Corps  must  employ  the  full  range  of  world-class 
integrated  instructional  and  information  technologies  that  enhance  the  full  spectrum  of 
Marine  Corps’  training  and  education.  Such  training  and  education  programs  must 
encompass  a  cradle-to-grave  methodology  that  begins  with  initial  entry  and  specialized 
skills  ( core  and  core  plus)  training  and  continues  through  career-long  professional 
development. 

Reviews  of  current  training  and  education  processes  reveal  problem  areas  that 
significantly  impact  on  operational  readiness  (Whitbeck,  1999).  First,  student  load 
frequently  exceeds  formal  school  seat  capacity  resulting  in  large  groups  of  Marines 
Awaiting  Training  (MAT).  Second,  initial  skill  resident  training  pipelines  are 
increasingly  long  and  have  increased  the  training  component  of  the  Transients,  Trainees, 
Patients  and  Prisoners  (T2P2)  manpower  account  and  decreased  overall  manning  in  the 
operating  forces.  Additionally,  career  progression  requirements  for  more  senior 
personnel  necessitate  time  away  from  operating  units  to  attend  appropriate  grade  level 
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schools,  which  also  add  to  the  T2P2  account.  The  current  focus  on  resident  formal  school 
attendance  as  a  prerequisite  for  Military  Occupational  Specialty  (MOS)  qualification  is 
currently  too  rigid  and  frequently  contributes  to  mismatch  problems  for  the  Marine  Corps 
Reserve  (Whitbeck,  1999).  Further,  the  current  Marine  Corps  Institute  (MCI)  paper- 
based  distance  learning  courses  are  not  closely  linked  to  resident  skill  training  curricula 
and  do  not  contribute  to  MOS  qualification  (Taylor,  2000).  Training  design  and 
development  is  often  fragmented,  lacking  an  integrated  MOS  curriculum.  Finally, 
existing  non-resident  PME  courses  are  based  primarily  on  resident  PME  curricula,  and 
similar  to  MCI  lessons  are  not  optimized  for  DL,  making  successful  course  completion 
difficult  (Jones,  1999). 

2.  Why  Distance  Learning? 

To  assist  the  Marine  Corps  in  meeting  these  challenges  with  increased  flexibility, 
the  Marine  Corps  Combat  Development  Command’s  Training  and  Education  Division 
introduced  the  Training  and  Modernization  Initiative  (TEMI)  in  FY  1998.  The  objective 
of  this  initiative  was  to  maximize  the  USMC’s  limited  training  design/development  and 
training  management  processes,  introducing  improved  technology  into  classrooms  and 
capitalizing  on  modem  DL  technologies.  PME  programs  will  also  benefit  from  this 
initiative  as  current  distance  education  courses  are  enhanced  through  the  application  of 
advanced/emerging  information  technologies. 

The  TEMI  attempts  to  correct  many  of  the  current  training  deficiencies  outlined 
above  through  a  comprehensive  review  and  restructuring  of  current  training  and 
education  processes.  Restructuring  will  include  improving  instructional  design, 
development,  delivery  and  management  processes.  DL  is  a  major  component  of  the 
TEMI.  DL  technology  has  the  potential  to  dramatically  change  the  way  in  which  Marines 
are  trained  and  educated  in  the  future.  Just  as  developments  in  weaponry  have  changed 
the  face  of  warfare,  investment  in  DL  technology  has  the  potential  to  transform  Marine 
Corps  training  and  education  from  a  centralized,  formal  school-based,  synchronous 
environment  to  a  more  distributed,  informal,  asynchronous  climate.  The  investment  in 
technology  will  be  driven  by  operational  readiness  requirements,  and  will  be  focused  on 
improving  the  efficacy  of  the  training  and  education  programs  provided  to  all  Marines. 

As  promising  as  these  initiatives  sound,  however,  their  implementation  comes  at  a 
price.  The  objective  of  this  research  is  to  analyze  the  feasibility  of  implementing  various 


DL  technologies  within  the  current  infrastructure  and  determine  if  a  positive  ROI  can  be 
obtained.  The  analysis  addresses  cost  benefit  techniques,  cost  factors  in  development  and 
cost  effectiveness  measures. 

3.  Why  ROI? 

According  to  Derryberry  (1998),  during  the  past  several  years  training 
organizations  have  been  pushed  by  their  clientele  -  both  external  and  internal  -  to  be 
more  accountable  for  the  quality  of  the  resources  and  services  provided.  This  focus  on 
accountability  has  reached  such  a  level  that  many  training  organizations  within  the 
government,  and  particularly  in  the  military,  are  finding  themselves  competing  against 
numerous  types  of  budgetary  decisions  to  maintain  their  current,  if  not  higher,  funding 
levels  (Lyau  &  Pucel,  1995).  At  the  same  time,  training  organizations  are  being  pushed 
to  offer  more  efficient  means  of  accessing  performance  improvement  resources  and 
services.  The  push  for  accountability  means  being  able  to  demonstrate  that  training 
interventions  have  a  direct  positive  impact  on  improving  organizational  competitiveness 
while  increasing  the  productivity  of  individual  contributors  (Derryberry,  1998).  This 
push  for  greater  efficiency  and  accessibility  has  understandably  led  to  expectations  that 
interactive  technologies  will  play  an  increasingly  significant  role  in  providing  both 
services  and  resources. 

Many  organizations  are  beginning  to  require  inclusion  of  a  cost/benefit  analysis 
(CBA)  or  ROI  analysis  to  justify  an  expenditure  such  as  what  is  required  for  a 
performance  improvement  environment  or  electronic  learning  system.  Simply  stated, 
ROI  methods  are  used  to  demonstrate  that  the  value  of  the  benefits  realized  from  using 
such  a  solution  is  going  to  exceed  the  price  of  its  development  and  implementation.  ROI 
and  technology  implementation  clearly  go  hand  in  hand  (Derryberry,  1998). 

Even  so,  the  author  suggests  that  ROI  is  not  simply  evaluation  methodology 
focused  solely  on  financial  concerns.  Technology  decisions  are  often  not  just  about  the 
financial  bottom  line.  Even  if  data  suggest  that  a  low-end  non-technological  solution 
may  yield  positive  results,  there  may  be  a  variety  of  organizational  reasons  that  a  low-end 
technology  solution  my  not  be  as  attractive  to  a  decision  maker  as  is  the  more 
sophisticated,  more  costly  technology  solution.  While  ROI  methods  provide  structure  for 
systematically  collecting,  organizing,  and  compiling  data,  the  interpretation  of  those  data 
and  the  resulting  perception  of  value  those  data  represent  may  be  highly  subjective. 
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Although  difficult  to  do,  a  sound  ROI  strategy  on  a  prospective  training  technology 
program  is  certainly  worth  the  effort  if  it  provides  the  justification  necessary  to  obtain 
funding  support.  In  today’s  highly  competitive  budgetary  environment,  such  a  strategy 
may  mean  the  difference  between  the  ultimate  success  or  failure  of  a  program. 

D.  RESEARCH  QUESTIONS 

1 .  What  are  the  current  DL  initiatives  currently  under  review  by  the  USMC? 

2.  What  are  the  DL  programs  currently  in  use  by  the  USMC? 

3.  What  is  the  ROI  objective  for  currently  USMC  training  programs? 

4.  What  are  the  costs  and  benefits  of  implementing  various  DL  applications  to 
current  training  programs  in  the  USMC? 

5.  Do  the  benefits  of  DL  initiatives  as  applied  to  these  training  programs 
outweigh  the  associated  costs? 

6.  What  are  some  common  training  development  cost  factors?  Should  they  be 
included  in  this  analysis? 

7.  Is  Cost-Benefit  Analysis  (CBA)  the  most  effective  method  for  determining 
ROI  on  DL? 

8.  How  cost-effective  are  current  training  programs?  Has  any  positive  ROI 
already  been  determined? 

9.  What  other  measures  of  cost-effectiveness  can  be  used  in  determining  ROI  for 
training? 

10.  What  aspects  of  quality  should  be  taken  into  account  when  determining  ROI 
for  training? 

E.  EXPECTED  BENEFITS  OF  THIS  THESIS 

This  study  will  provide  information  to  determine  the  feasibility,  in  terms  of  costs 
and  benefits,  of  various  Distance  Learning  initiatives  currently  under  review  by  the  U.S 
Marine  Corps.  It  will  serve  as  a  decision  tool  by  illustrating  the  potential  ROI  associated 
with  each  of  these  programs,  based  on  the  stated  assumptions. 
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II.  USMC  DISTANCE  LEARNING  PROGRAMS 

A.  DISTANCE  LEARNING  DEFINED 


For  the  purposes  of  this  paper,  the  U.S.  Marine  Corps  has  defined  Distance 
Learning  (DL)  as:  “structured  learning  that  takes  place  without  the  physical  presence  of 
the  instructor.”  (HQMC  T&E  DL  ROADMAP,  1999).  This  definition  comprises 
traditional  paper-based  correspondence,  but  not  stand-alone  modeling  and  simulation. 

Key  attributes  of  a  sound  DL  program  include  (Portaway  &  Lane,  1992): 

•  Physical  distance  between  learner  and  instructor 

•  Program  sponsored  by  an  academic  institution  or  functional  organization 

•  Program  is  part  of  a  structured  curriculum  with  stated  objectives 

•  Program  provides  for  two-way  communication  and  feedback  between 
institution  and  learner 

•  Program  deployed  outside  the  confines  of  a  resident  schoolhouse  or  campus 

•  Program  includes  processes  to  evaluate  learning  outcomes 

The  traditional  classroom  may  be  viewed  as  an  island  on  which  a  teacher  and  a 
group  of  students,  supplied  with  textbooks  and  other  resources,  manage  the  educational 
process.  Occasionally,  the  students  and  teachers  take  trips  into  the  outside  world,  but  the 
overall  educational  process  primarily  focuses  inward  on  things  happening  in  the 
classroom.  Contemporary  telecommunications  technology  can  invert  this  traditional 
focus.  Classrooms  now  orient  to  the  world  outside  rather  than  the  immediate 
environment.  Instead  of  remaining  isolated  islands,  classrooms  can  now  be  linked  by 
communication  highways  transmitting  data  (video  and  audio)  to  multitudes  of  remote 
sites.  Teachers  and  students  alike  may  now  travel  on  this  new  information  highway  with 
easy  access  to  vast  databases  and  jointly  participate  in  activities  that  involve  other 
students  in  other  countries  (Lynton,  1992).  This  new  technology  can  provide  educational 
opportunities  to  students  who  lack  access  to  a  capable,  resident  classroom  environment. 
Discriminating  factors  such  as  geographic  dispersion,  temporary  separation,  or 
unavailability  of  locally  qualified  instructors  need  no  longer  affect  the  individual’s  access 
to  a  quality  education.  Distance  learning  is  an  exceptional  alternative  when  a  training 
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facility  cannot  feasibly  or  economically  provide  quality  face-to-face  instruction  to  serve 
the  remote  student’s  needs. 

Bramble  (1990)  states  that  distance  learning  is  by  no  means  a  panacea  for  all  the 
ills  of  the  educational  process,  nor  should  it  be  viewed  as  a  permanent  replacement  for 
face-to-face  instruction.  He  found  that  students  actually  prefer  face-to-face  instruction. 
Unfortunately,  distance  learning  is  often  viewed  as  a  conceptual  threat  to  the  proponents 
of  conventional  educational  methods.  As  such,  distance  learning  advocates  must  be 
careful  to  present  their  approach  as  an  alternative  to  conventional  instruction,  to  be  used 
when  conventional  methods  are  not  feasible  or  economical.  As  this  study  will 
demonstrate,  distance  learning  can  be  a  cost  effective  alternative,  although  proponents 
should  be  cautious  not  to  promote  distance  education  solely  on  the  basis  of  low  cost.  To 
properly  support  this  emerging  environment,  which  presently  seems  to  offer  great 
potential,  organizations  must  first  establish  an  infrastructure  of  policies  and  standards 
focused  on  long-term  strategies  for  efficient  implementation  of  distance  education. 

B.  USMC  DL  TECHNOLOGY  STRATEGY 

Distance  Learning  can  be  accomplished  through  a  variety  of  media  including 
paper-based  instruction,  compressed  video,  Electronic  Performance  Support  Systems 
(EPSS)  and  Virtual  Reality  (VR)  simulations.  As  previously  mentioned,  DL  and  resident 
courses  are  viewed  as  complementary  forms  of  instruction  that  enable  the  delivery  of  the 
right  mix  of  instructional  methods  at  the  right  time.  In  order  to  successfully  transition  to 
an  effective  DL  program,  however,  the  organization/institution  must  fully  embrace  an 
individual  learning  model  vice  merely  shifting  to  a  technique  of  remote  teaching. 
Properly  implemented,  this  transition  will  have  wide  reaching  implications  across  any 
organization  that  adopts  it,  and  will  require  a  fundamental  shift  in  traditional  approaches 
to  training  and  education. 

The  USMC’s  technology  strategy  is  centered  on  the  learner.  The  intent  is  to 
develop  systems  that  enable  current  educational  systems  to  reach  more  Marines  with 
better  and  more  focused  training  and  education  programs  in  the  near  future.  In  keeping 
with  the  Commandant’s  Planning  Guidance  (Krulak  1995),  the  aim  of  Marine  Corps’ 
training  and  education  is  to  extend  learning  beyond  the  boundaries  of  the  traditional 
classroom  by  exploiting  technology.  Technology  has  matured  to  the  point  whereby  it  is 
now  possible  to  deliver  high  quality  interactive  instructional  materials  using  multiple 
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media  formats  without  regard  to  time,  space  or  distance.  Moreover,  the  rapid  growth  of 
the  Internet  has  changed  the  perception  of  information  sharing,  electronic  commerce, 
methods  of  instructional  delivery  and  collaborative  learning.  Academia,  government  and 
industry  are  using  this  medium  to  train  and  educate  the  modem  work  force.  The  Marine 
Corps  will  also  use  Internet-based  technologies  to  deliver  interactive  learning  products 
and  facilitate  a  collaborative  learning  environment  in  the  future. 

Distance  Learning  programs  are  learner-centered  rather  than  instructor-centered. 
They  are  being  designed  specifically  with  the  distance  learner  in  mind  as  per  the  HQMC 
Training  &  Education  DL  Roadmap.  In  HQMC’s  plan  DL  solutions  will  be  network- 
delivered  where  possible;  exploiting  the  power  of  the  Internet  in  order  to  distribute  and 
track  computer-based  interactive  multimedia  instruction  (IMI).  Realizing  that  a  robust 
infrastructure  is  required  to  support  a  network-based  learning  environment,  interim 
hybrid  solutions  are  already  in  the  works.  The  roadmap  further  outlines  solutions  that 
include  a  combination  of  the  Internet,  CD-ROM  technology  and  traditional  paper-based 
DL  courses,  until  the  proposed  enterprise  network  can  adequately  support  network 
delivery.  The  plan  also  describes  how  this  network-based  learning  will  be  additionally 
supported  with  video  teleconferencing  and  teletraining  capabilities  to  leverage  existing 
service  and  government  video  networks. 

Finally,  for  the  long  term  the  HQMC  DL  Roadmap  identifies  the  development  of 
artificial  intelligence,  intelligence  agents,  intelligent  tutoring  systems,  performance 
support  and  embedded  training  technologies  that  will  be  encouraged  during  new  system 
acquisitions  to  reduce  the  requirement  for  equipment  specific  training. 

C.  USMC  DISTANCE  LEARNING  STRATEGIC  VISION 

Existing  and  emerging  information  technologies  provide  an  excellent  opportunity 
to  deliver  flexible  and  adaptable  training  and  education  solutions.  The  goal  of  USMC 
training  and  education  programs  is  to  encompass  a  “cradle  to  grave”  approach  that  begins 
with  initial  entry  and  specialized  skills  training  and  continues  through  career-long 
professional  development  (HQMC  T&E  DL  ROADMAP  1999). 

The  Marine  Corps  developed  the  Training  and  Education  Modernization 
Initiative  ( TEMI)  as  a  substantial  part  of  its  overall  educational  strategy  to  meet  new 
challenges  with  increased  flexibility.  The  objective  of  this  initiative  is  to  maximize  the 
Corps’  limited  training  and  education  resources  by  restructuring  current  institutional 
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training,  improving  existing  training  design/development  and  training  management 
processes,  while  simultaneously  introducing  technology  improvements  into  classrooms 
that  capitalize  on  modem  DL  technologies.  Professional  Military  Education  (PME) 
programs  will  also  benefit  from  this  initiative  as  current  distance  education  courses  are 
enhanced  through  the  application  of  advanced/emerging  information  technologies. 

D.  OPPORTUNITIES  AND  CHALLENGES 

The  question  naturally  arises  as  to  why  DL  and  training  technologies  should  be 
embraced?  An  increased  emphasis  on  the  use  of  DL  technologies  will  greatly  enhance 
training  and  education  opportunities  for  all  Marines  for  a  number  of  reasons.  First,  there 
is  a  momentum  across  industry,  academia  and  the  other  services  to  exploit  modem 
technology  for  the  purpose  of  improving  instruction  and  increasing  opportunities  for 
access.  Recent  advances  in  technology  have  made  DL  an  increasingly  viable  option  to 
deliver  courses  of  instruction  on  demand,  with  minimal  regard  to  time  and  distance. 
Secondly,  the  Executive  Branch  has  specifically  tasked  the  services  as  well  as  all  other 
Federal  agencies  to  increase  the  use  of  DL  where  possible  to  reduce  the  cost  of 
institutional  training  and  education  programs  (EO  13111,  1999).  Finally,  DL  has  the 
potential  to  “level  the  playing  field”  for  reserve  forces  by  providing  them  with  increased 
opportunities  for  training  and  education.  Expanded  use  of  DL  will  help  reduce  resident 
training  time  and  training  and  education  costs,  while  increasing  educational  opportunities 
for  all  Marines  (Tyler,  1998). 

The  military  and  civilian  industry  can  no  longer  afford  to  conduct  training  and 
education  in  the  same  outmoded,  predictable  fashion.  The  challenge  today  is  to  meet 
increased  training  and  education  demands  in  spite  of  reductions  in  funding,  manpower 
and  training  infrastructure.  While  resources  are  decreasing,  requirements  for  training  and 
education  are  constantly  increasing.  Changes  in  demographics,  procurement  practices 
and  organizational  structure  have  resulted  in  a  younger  force,  new  equipment,  changing 
missions  and  MOS  mergers;  all  of  which  impact  training  and  education  requirements. 
Studies  in  both  civilian  academic  and  military  training  environments  assert  that  DL  offers 
a  means  to  meet  these  emerging  and  fluctuating  demands  (Moore  &  Kearsky,  1996; 
Freeman,  1999). 

Distance  Learning  applications  can  reduce  the  cost  of  traditional  training  and 
education  while  still  meeting  traditional  training  and  education  needs  (OUSDP&R, 
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1999).  A  study  conducted  by  the  Logistics  Management  Institute  (Belcher,  1999)  states 
that  use  of  network  and  CD-ROM  distribution  in  DL  can  significantly  reduce  the 
production  and  distribution  costs  associated  with  conventional  paper-based  courses. 
Phelps,  Wells,  Ashworth  and  Hahn,  (1992)  convincingly  demonstrate  how  offering  a 
course  via  DL  can  also  reduce  or  eliminate  travel  and  per  diem  costs.  An  analysis  of  two 
identical  study  groups,  one  in  residence  and  the  other  using  Computer-mediated 
Communications  (CMC)  technology  from  the  U.S.  Army’s  Engineer  Officer  Advanced 
Course,  resulted  in  cost  savings  in  excess  of  $600K  based  on  five  iterations  of  the  course. 
Course  topics  included  technical  engineering  skills  as  well  as  Army  tactics,  leadership 
and  briefing  and  presentation  skills.  The  Battle  Staff  Noncommissioned  Officers’  Course 
(BSNCOC)  at  Fort  Bliss,  Texas,  estimates  an  average  cost  savings  of  more  than  $30K  per 
student  for  its  six-week  course  (Belcher,  1999).  Adopting  DL  practices  can  also  provide 
widespread  access  to  training  and  education  resources.  For  example,  it  can  increase 
student  throughput  as  illustrated  by  the  Air  Force  Institute  of  Technology  (AFIT),  which 
increased  student  throughput  from  300  to  3000  for  its  Acquisition  Planning  and  Analysis 
course  by  converting  it  to  a  DL  format  (IDEA,  1999).  The  Naval  Postgraduate  School  is 
pursuing  a  similar  course  of  action  for  several  of  its  graduate  education  programs 
(Hazard,  1999).  Finally,  the  Army  National  Guard  is  developing  comprehensive  DL 
programs  to  increase  training  opportunities  for  over  36,000  Guardsman  across  the  United 
States  (Metzko,  1996). 

Advanced  technologies  are  currently  being  explored  to  develop  and  deliver 
learning  products  just  in  time,  when  and  where  they  are  most  needed.  In  the  future, 
Marines  can  expect  to  use  a  Navy/Marine  Corps  Intranet  (N/MCI),  the  Internet,  learning 
resource  centers  (LRCs),  interactive  multimedia  instruction  (IMI),  video  teletraining 
(VTT)  and  embedded  training  (ET)  to  master  new  skills.  Although  Distance  Learning 
technologies  have  the  potential  to  yield  savings,  they  can  be  costly  to  initiate,  requiring  a 
substantial  up-front  investment  (Tyler,  1999).  The  cost  of  implementing  technology  that 
fails  to  live  up  to  expectations,  highlights  the  importance  of  a  coordinated  approach  to 
research,  development,  acquisition  and  life  cycle  management.  Acquisition  projects,  such 
as  the  USMC’s  DL  Program  are  therefore  consulting  training  design  specialists  early  in 
the  requirements  definition  phase  to  reduce  the  cost  of  training  and  align  it  with  the 
existing  Marine  Corps  training  standards  (MCSC  DL  LCCE,  1998).  The  continued 
partnership  between  the  acquisition  force,  manpower  specialists  and  training  has  the 
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potential  to  achieve  significant  life  cycle  cost  efficiencies  for  major  Marine  Corps 
equipment  procurements  in  the  future. 

Many  of  these  technologies  are  still  under  development  and  require  further  study 
to  ensure  that  each  will  be  a  cost-effective  solution  to  growing  training  and  education 
requirements.  Likewise,  the  partnership  between  acquisition,  manpower  and  training 
remains  in  its  infancy  and  must  be  cultivated  to  fruition.  Moreover,  development  and 
maintenance  of  long-term  partnerships  with  other  services  remain  crucial  to  benefit  from 
their  technological  advances,  while  meeting  Marine  Corps  specific  requirements  and 
defraying  overall  costs. 

E.  MODERNIZATION  TENANTS 

Training  and  education  is  an  investment  in  operational  readiness.  Restructuring 
existing  programs  must  focus  on  enhancing  operational  readiness  by  making  training  and 
education  better  and  more  efficient.  The  basic  tenets  of  the  modernization  initiative  are 
as  follows  (HQMC  T&E  DL  ROADMAP,  1999): 

•  Training  and  education  are  a  core  responsibility  of  the  Service 

•  Operational  readiness  is  the  primary  consideration  for  implementing  training 
and  education  programs 

•  Marine  Corps  training  and  education  will  be  standards-based 

•  A  Marines’  educational  experience  is  part  of  a  career-long  learning  continuum 
supporting  the  operational  needs  of  the  total  force 

•  Technology  will  be  leveraged  to  improve  the  effectiveness  and  efficiency  of 
Marine  Corps  training  and  education 

•  DL  will  increasingly  be  used  to  meet  future  Marine  Corps  training  and 
education  requirements 

•  The  Marine  Corps  will  leverage  other  DoD  and  governmental  agency  DL  and 
instructional  technology  efforts 

F.  THE  MARINE  CORPS  DISTANCE  LEARNING  PROGRAM 

1.  Overview 

The  revised  Marine  Corps  DL  program  expands  on  the  paper-based  system  of  the 
Marine  Corps  Institute  (MCI)  by  developing  integrated  training  and  education  programs 
for  initial  skill  and  skill  progression  training  that  lead  to  Military  Occupational  Specialty 
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(MOS)  qualification,  and  also  enhance  current  distance  education  PME  programs.  The 
DL  program  is  a  funded  Total  Force  program  supporting  both  the  active  Marine  Corps 
and  the  Marine  Corps  Reserve.  Implementing  comprehensive  DL  programs  is  a  complex 
undertaking  and  involves  developing  innovative  solutions  in  four  major  technology 
domains:  instructional  content,  delivery  infrastructure,  instructional  management  systems 
for  delivering  courses  and  managing  students  in  a  distributed  environment  and  instmctor 
training  and  support  (HQMC  T&E  DL  ROADMAP,  1999).  The  DL  program  will  offer 
solutions  in  each  technology  domain  and  will  be  implemented  according  to  a  two-phase 
schedule  -  a  pilot  phase  (FY97-99)  and  a  program  expansion  phase  (FY00-05).  The  pilot 
effort  was  initiated  to  study  infrastructure  and  process  and  resource  requirements,  to 
successfully  establish  a  viable  DL  program.  Lessons  learned  from  the  pilot  effort  are 
being  used  to  shape  the  DL  program  so  that  limited  resources  can  be  effectively  focused 
on  establishing  a  flexible  Corps-wide  DL  capability  in  the  future.  A  major  part  of 
implementing  the  USMC  DL  program  will  center  on  the  establishment  and  activation  of 
the  Marine  Corps  Learning  Network  or  MarineNet  infrastructure.  As  of  this  writing,  it  is 
envisioned  that  MarineNet  will  function  as  a  subcomponent  of  the  larger  N/MCI  that  is 
currently  being  planned  (CNO,  2000;  Whitbeck,  2000).  For  the  purposes  of  this  paper, 
the  use  of  the  terms  N/MCI  and  MarineNet  will  be  synonymous.  MarineNet  is  a  wide- 
ranging  initiative  that  provides  the  supporting  infrastructure  and  access  points  to  enable 
the  delivery  of  high  quality  training  and  education  to  all  Marines,  regardless  of  location. 
MarineNet  is  essentially  a  Marine  Corps-wide  distributed  Intranet  supported  by 
compressed  video  technologies  that  will  enable  Marines  and  civilian  DoD  personnel  to 
learn  via  the  appropriate  media  when  and  where  learning  is  most  needed  and  required 
(HQMC  T&E  DL  ROADMAP,  1999).  MarineNet  is  composed  of  electronic  interactive 
DL  courseware,  hardware,  software  and  network  components  necessary  to  distribute 
electronic  instruction  over  Marine  Corps  wide  area,  metropolitan  area  and  local  area 
networks. 

The  N/MCI  will  use  the  planned  Defense  Information  Infrastructure  (DII) 
architecture  for  wide  area  connectivity  across  the  Marine  Corps  enterprise,  and  the 
planned  Base  Telecommunications  Infrastructure  (BTI)  upgrades  for  metropolitan  area 
and  local  area  network  connectivity  aboard  bases  and  stations  (HQMC  T&E  ROADMAP, 
1999).  The  R-net  will  provide  network  connectivity  to  the  194  reserve  sites  across  the 
country  (Metzko,  1996).  Open  system  architecture  will  be  used  to  ensure  interoperability 
between  Marine  Corps  and  other  service  DL  and  communications  systems  and  platforms. 
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2.  Organizational  Roles  to  Support  USMC  DL 

The  DL  program  and  MarineNet  will  be  supported  by  the  following 
organizational  components:  the  Distance  Learning  Center  (DLC),  Functional  Learning 
Centers  (FLCs)  and  Area  Learning  Centers  (ALCs).  These  components  conform  to  a 
three-tier  approach  for  DL.  Each  center  corresponds  to  an  echelon  within  the  Marine 
Corps  and  has  specific  roles  and  responsibilities  to  that  organizational  level.  As 
mentioned  previously,  the  components  of  the  functional  structure  are  physically 
interconnected  via  the  BTI.  The  BTI  will  provide  an  upgraded  network  infrastructure 
aboard  all  bases  and  stations  as  well  as  telecommunications  “pipes”  between  Marine 
locations  via  the  Defense  Information  Systems  Network  (DISN)  (HQMC  T&E  DL 
ROADMAP,  1999). 

The  MCI  designated  as  the  DLC,  is  the  Corps-level  organization  that  will  provide 
Marine  Corps-wide  standardization,  certification  and  quality  control  for  all  DL  in  the 
future.  The  DLC  will  provide  a  consolidated  Corps-wide  on-line  catalog  of  DL  products 
to  include  a  VTT  broadcast  listing,  accessible  through  the  Internet.  The  DLC  will  also 
manage  master  DL  manpower  data,  ensuring  complete  and  accurate  information  on 
enrollments,  completions  and  qualifications  are  passed  on  to  the  Marine  Corps  Total 
Force  System  (MCTFS). 

The  formal  schools  currently  in  existence  will  serve  as  FLCs  and  functional  area 
proponents,  managing  all  electronic  DL  courses  related  to  their  specific  area  of  expertise. 
Multimedia  course  development  and  maintenance  will  occur  at  Regional  Development 
Centers  (RDCs).  RDCs  are  supporting  organizations  that  will  also  manage  contractor- 
developed  DL  products,  ensuring  that  they  conform  to  established  Marine  Corps’ 
standards  and  protocols.  RDCs  will  be  under  the  operational  control  of  the  DLC  and 
under  the  administrative  control  of  the  local  base  establishment.  Electronic  course 
distribution  will  occur  from  FLC  servers,  while  paper-based  distribution  functions  will  be 
retained  by  the  MCI. 

The  RDCs  will  provide  centrally  managed  and  dedicated  course  development 
teams  at  selected  Marine  Corps  Schools  (i.e.,  FLCs).  The  two  FLCs  initially  selected  to 
host  RDCs  are  the  Marine  Corps  Communications  and  Electronics  School,  located  at  29 
Palms,  California,  and  the  Marine  Combat  Service  Support  Schools,  Camp  Lejeune, 
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North  Carolina.  The  RDCs  will  support  the  FLCs  in  their  respective  geographic  locations 
(i.e.,  East  Coast/West  Coast).  Each  RDC  will  be  responsible  for  developing  technology- 
based  courseware  and  managing  contractor  developed  content  for  its  client  FLCs.  The 
FLCs  will  provide  actual  course  content  and  subject  matter  expertise.  The  course 
development  section  of  the  MCI  will  provide  content  development  support  to  smaller 
formal  schools/detachments,  as  well  as  assist  with  overflow  course  development  from  the 
two  RDCs.  Establishing  an  RDC  for  PME  course  development  at  the  Marine  Corps 
University  is  scheduled  to  begin  in  FY  00. 

The  ALC  is  the  primary  metropolitan  area  network  for  delivering  DL  courseware 
to  Marines  in  a  given  geographic  area.  The  ALC  will  be  comprised  of  a  Training  and 
Education  Point  of  Presence  (TEPOP)  server  with  one  or  more  interconnected  Learning 
Resource  Centers  (LRCs)  and  VTT  Centers.  The  number  of  LRCs  and  VTT  Centers 
within  the  ALC  geographic  region  will  depend  upon  the  size  of  the  region  and  the 
population  size  supported  by  the  base,  station  or  site.  The  ALC  will  provide  local  DL 
account  management,  storage  and  distribution  of  electronic  courseware  to  any  authorized 
user  connected  to  the  ALC  network. 

The  primary  access  to  MarineNet  will  occur  at  the  local  level  ALCs.  The  ALC 
TEPOP  server  suite  is  designed  to  provide  local  (regional)  instructional  material  storage, 
distribution  and  security  services.  One  TEPOP  is  required  per  ALC  (region)  and  will 
service  many  “tenant”  FLCs  and  LRCs.  The  storage  resource  will  provide  the  capacity 
for  all  electronic  training  material  to  be  accessed  through  workstations  aboard  the  base,  as 
well  as  the  necessary  management  tools  to  monitor  student  progress,  monitor  network 
utilization,  determine  courseware  availability  and  suitability  and  maintain  statistical 
information. 

The  LRC  is  designated  as  the  primary  location  to  access  DL  courseware  for  those 
Marines  who  do  not  have  access  to  computer  workstations.  The  LRC  is  a  client-server 
Local  Area  Network  (LAN)  system  connected  to  the  base  network  backbone  and  accesses 
courseware  stored  on  the  regional  TEPOP  server  suite.  Each  LRC  can  accommodate 
approximately  25  to  40  simultaneous  users.  Approximately  94  LRCs  will  be  procured  as 
part  of  the  DL  program  in  addition  to  the  workstations  already  connected  to  the  Marine 
Corps  enterprise  network  (HQMC  T&E  DL  ROADMAP,  1999). 

Where  established,  VTT  Centers  will  provide  the  capability  to  conduct  DL  using 
the  latest  Video  Teleconferencing  (VTC)  technologies.  The  centers  will  have  a  two-way 
video  and  two-way  audio  (2V/2A)  capability  and  the  ability  to  accommodate  15  to  20 
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simultaneous  users.  The  system  will  conform  to  all  DoD  standards  and  guidelines  and 
will  be  capable  of  multi-point  conferencing  with  all  VTT  centers  DoD-wide  (Cabrera, 
1999).  The  MarineNet  VTT  systems  will  leverage  the  existing  Marine  Corps  Satellite 
Education  Network  (MCSEN),  and  the  Navy’s  CNET  Electronic  Schoolhouse  Network 
(CESN)  capabilities.  The  DL  program  will  field  25  additional  VTT  sites  that  will  be 
interconnected  with  the  MCSEN  and  CESN  systems  at  both  active  and  reserve  locations. 

Plans  also  call  for  deployable  LRCs  that  will  provide  operational  units  with  the 
capability  to  access  DL  resources  while  deployed  abroad  (HQMC  T&E  DL  ROADMAP, 
1999).  These  small  self-contained,  ruggedized  client-server  networks  will  emulate  the 
capability  of  the  fixed  site  LRC.  Further,  they  will  have  the  capability  to  connect  to 
shipboard  or  external  Transmission  Control  Protocol/Intemet  Protocol  (TCP/IP) 
networks.  The  system  is  composed  of  a  server  and  ten  client  workstations.  Courseware 
will  be  uploaded  onto  the  deployable  server  prior  to  deployment  and  updated  through  a 
“reach  back”  capability  to  the  host  unit  TEPOP  where  adequate  long  haul 
communications  links  exist. 

The  Marine  Corps  DL  program  relies  upon  the  implementation  of  the  BTI 
upgrade.  Delays  in  BTI  network  implementation  will  thus  defer  the  fielding  of 
MarineNet  (Tyler,  1998).  Program  sponsors  are  currently  working  closely  with  the 
Marine  Corps  Systems  Command  and  the  bases/stations  to  coordinate  BTI 
implementation  with  MarineNet  fielding  (Tyler,  1998).  Successful  integration  with  the 
R-Net  is  also  critical  to  the  success  of  the  DL  program  for  the  Reserve  component  (Tyler, 
1998).  R-net  integration  issues  will  be  thoroughly  studied  during  the  pilot  phase  of  the 
DL  initiative  to  ensure  adequate  interoperability  before  program  expansion  in  FY  00. 

3.  DL  Content  Development  and  Management  Issues 

An  integral  part  of  the  Marine  Corps  TEMI  also  involves  a  comprehensive  review 
of  all  formal  training  tacks.  The  Training  and  Education  Division  will  lead  the  effort  to 
conduct  detailed  training  reviews  for  100  percent  of  all  institutional  training  courses  over 
the  next  six  years  (HQMC  DL  ROADMAP,  1999).  Training  reviews  will  commence 
with  those  occupational  fields  determined  to  have  the  greatest  potential  savings  in  terms 
of  MAT  and  transient  times.  Further,  those  courses  with  insufficient  capacity  to  meet 
throughput  requirements  will  be  the  first  candidates  for  review.  The  objectives  of  the 
training  review  process  are: 
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(1)  Design  a  comprehensive  MOS  training  concept. 

(2)  Identify  essential  job  performance  competencies  and  supporting  tasks  for 
MOS  qualification. 

(3)  Design  a  training  progression  model  that  meets  career  track  requirements  for 
each  MOS. 

(4)  Redesign  courses  as  appropriate  to  incorporate  both  resident  and  DL  to 
achieve  MOS  qualification  within  established  resource  guidelines. 

Once  the  curriculum  reviews  are  completed,  FLCs  will  design  and  develop  both 
resident  instruction  and  DL  products  to  support  the  new  training  progression  models  and 
curricula.  Course  development  efforts  at  the  formal  schools  will  be  supported  by  the 
RDC  and  MCI  development  teams,  or  through  commercial  outsourcing.  The 
implementation  of  revised  integrated  curricula  will  vary  depending  upon  the  complexity 
and  subject  matter  involved;  however,  the  time  window  for  this  process  is  to  begin 
delivery  of  new  training  and  education  solutions  within  six  to  nine  months  of 
commencing  the  review  process. 

Another  component  of  the  DL  program  is  an  enterprise-wide  management  system 
to  coordinate  courses  and  students  in  a  distributed  environment.  The  system  must  be 
flexible  to  adequately  support  a  highly  mobile  population  of  active  duty  and  reserve 
Marines,  to  include  civil  servants  for  on-line  registration,  tracking  and  assessment. 
Further,  the  system  must  be  able  to  accommodate  not  only  training  events,  but  PME  and 
voluntary  education  opportunities  as  well.  The  system  of  the  future  will  provide  an 
alternative  registration  and  assessment  capability  for  traditional  DL  courses  until  they  are 
eventually  phased  out,  and  serve  as  a  secure  gateway  for  new  on-line  DL  courseware  and 
materials.  The  system  is  planned  to  be  accessible  through  both  MarineNet  and  the 
conventional  Internet. 

From  a  human  resource  perspective,  a  further  consideration  is  that  DL  is  a 
relatively  new  operating  environment  for  Marine  Corps  instructors  and  curriculum 
developers.  Exposing  these  instructors,  training  developers  and  senior  leaders  to 
emerging  DL  instructional  technologies  will  greatly  assist  them  in  overcoming  the  normal 
apprehension  associated  with  applying  new  methods  and  training  concepts.  Accordingly, 
instructor  development  and  support  are  critical  factors  to  the  overall  success  of  the  DL 
program.  To  help  facilitate  this  transition,  the  DLC  and  the  Instructional  Management 
Schools  located  on  both  coasts  will  develop  instructor-training  programs  and  provide 
technical  support  to  ease  the  expansion  of  DL  capabilities  across  all  Marine  Corps  formal 
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schools.  Training  programs  will  include  information  on  the  instructional  design  process, 
implementation  of  DL  instructional  technologies  and  project  management.  Additionally, 
the  DLC  will  provide  a  help  desk  to  support  DL  instructors  and  curriculum  developers  at 
the  RDCs  and  FLCs. 

This  review  process  is  a  significant  effort.  It  will  require  time  and  dedicated 
effort  on  the  part  of  all  relevant  parties  to  develop  pertinent  and  cost  effective  training 
solutions  for  each  of  the  MOSs  considered.  This  situation  does  however,  present  the 
Marine  Corps  with  an  opportunity  to  enhance  training  and  education  through  a 
restructuring  effort  that  will  yield  even  greater  dividends  in  the  future. 

G.  CHAPTER  SUMMARY 

The  objective  of  the  TEMI  and  more  specifically  the  implementation  of  DL,  is  to 
provide  better  and  more  effective  training  in  an  environment  of  increasingly  fewer 
resources.  Initiating  modem  instructional  technologies  to  both  deliver  DL  and 
simultaneously  enhance  our  resident  school  training,  presents  significant  but  not 
insurmountable  challenges.  Meeting  the  established  goals  of  reducing  MAT,  overall 
training  time  and  training  structure,  while  providing  better  and  more  efficiently  delivered 
instruction  to  Marines  is  only  a  part  of  the  challenge.  Shifting  to  a  learner-centric  vice 
instructor-centric  approach  to  learning,  while  ensuring  computer  literacy,  making  limited 
time  available  for  Marines  to  train  via  DL  techniques,  and  also  overcoming  institutional 
bias  and  resistance  to  change,  present  challenges  of  equal  if  not  greater  magnitude. 

Distance  Learning,  although  not  a  panacea  for  all  training  and  education  issues,  is 
an  established  and  proven  method  for  delivering  training  and  education  without 
sacrificing  quality.  MCI  has  been  providing  paper-based  DL  to  Marines  for  more  than  75 
years  with  great  success.  Modem  technology  provides  an  opportunity  to  significantly 
improve  the  quality  of  the  learning  experience.  Further,  such  technology  will  enable  the 
Marine  Corps  to  successfully  achieve  its  goal  of  providing  more  Marines  a  higher  quality 
of  training  and  education  while  reducing  the  required  resources. 
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III.  ELEMENTS  OF  ROI  FOR  DISTANCE  LEARNING 


A.  INTRODUCTION 

Literature  about  the  cost  effectiveness  of  distance  education  tends  to  focus  on 
costs  of  specific  media  implementations  (Rumble  1989;  Solomin  and  Holden  1988); 
pragmatic  discussions  of  models  and  costing  issues  (Markowitz  1987;  Rumble  1988);  and 
comparisons  between  traditional  resident  instruction  and  distance  education  (Hahn, 
Ashworth,  Phelps,  Wells,  Richards  &  Daveline  1991;  Laidlaw  and  Layard  1974;  Muta 
1985;  Rule,  DeWulf  &  Stowitschek  1988;  Wagner  1977).  The  literature  suggests  that 
distance  education  can  be  less  expensive  than  resident  instruction,  depending  upon 
student  enrollment  and  the  fixed  costs  of  course  development  and  delivery.  Furthermore, 
the  literature  compares  the  training  effectiveness  of  distance  education  to  face-to-face 
resident  instruction,  and  suggests  that  distance  courses  delivered  with  various  media 
compare  favorably  with  resident  instruction.  The  conclusion  that  such  effectiveness  is 
equal  among  these  two  methods  is  not  generally  accepted,  however,  due  to  disagreement 
over  such  issues  as  the  value  of  human  interaction  in  training  and  education  (Beare  1989; 
Jevons  1982;  Misanchuk  1982;  Whittington  1987). 

The  major  challenge  of  distance  education  is  increasing  access  while  controlling 
the  cost  of  delivery.  Meeting  this  challenge  while  maintaining  effective  instruction  is 
crucial  to  achieving  the  return  on  investment  necessary  to  insure  the  viability  of  distance 
learning  programs.  The  imperative  of  access  and  cost  effectiveness  requires  the  selection 
of  the  least  expensive  alternative  that  meets  the  course  objectives  and  reaches  the 
intended  audience.  All  things  being  equal,  the  more  access  and  less  expensive  the 
method,  the  better  for  all  concerned.  Courseware  designers  and  managers  are  expected  to 
be  good  stewards  of  resources  entrusted  to  them  seeking  efficiency  in  their  designs. 
Executive  Order  13111  recognized  an  obligation  to  increase  access  while  controlling 
costs. 

It  would  appear  that  for  all  the  emphasis  placed  on  minimizing  training  expenses, 
a  thorough  cost  analysis  of  training  alternatives  is  not  an  easy  thing  to  do,  particularly 
where  DL  technology  is  concerned.  Variables  can  be  many  and  ill  defined.  The 
remainder  of  this  chapter  provides  insight  to  those  elements  critical  in  obtaining  a 
favorable  ROI  for  DL.  Issues  include  a  discussion  of  cost  benefit  analysis  (CBA)  versus 
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cost  effectiveness  analysis  (CEA),  a  review  of  cost-benefit  techniques,  factors  of  cost 
analysis  in  training  and  development,  and  a  look  at  measures  of  effectiveness  commonly 
used  in  determining  ROI  for  DL. 

B.  ISSUES  AFFECTING  ROI  FOR  DL 


1.  Overview 

If  one  can  deliver  instruction  to  many  students  at  their  home  stations 
economically,  there  surely  must  be  cost  savings  over  bringing  those  same  students  to 
centralized  training.  Russell  (1999)  argued  that  DL  can  reduce  training  travel  and  per 
diem  costs  compared  to  schoolhouse-based  instruction,  and  that  the  training  results  in  no 
significant  difference  in  training  effectiveness.  This  reasoning,  although  supported  by 
over  300  studies,  is  still  considered  highly  subjective  and  is  far  from  conclusive. 
Discussion  comparing  the  effectiveness  of  these  training  methods  is  a  very  complex 
issue,  and  is  beyond  the  scope  of  this  study.  To  avoid  confusion  in  the  remainder  of  this 
paper,  the  context  of  effectiveness  will  refer  to  cost  analysis  only. 

Distance  Learning  involves  more  than  technology.  It  is  a  combination  of  people, 
process  and  technology  that  creates  effective  and  efficient  DL  programs  (Kidwell  1998). 
There  is  an  underlying  study  of  pedagogy  that  goes  into  determining  what  DL  option,  if 
any,  makes  sense  for  a  particular  course.  Most  DoD  and  service  DL  strategies  recognize 
this  requirement  that  training  be  student-centered,  and  not  technology-centered. 
Questions  remain,  how  should  the  military  services  decide  whether  to  invest  in  DL 
programs?  How  should  they  determine  in  which  technologies  to  invest?  How  should  the 
DoD  measure  the  value  of  those  investments?  What  costs  and  benefits  are  relevant?  The 
following  review  may  help  the  reader  understand  the  factors  that  contribute  to  arriving  at 
a  decision  as  to  the  applicability  of  DL  solutions  in  a  learning  environment. 

2.  Synchronous  versus  Asynchronous  Delivery 

Distance  education  delivery  systems  are  categorized  as  either  synchronous  or 
asynchronous.  Synchronous  delivery  requires  instructors  and  students  to  participate  at 
the  same  time,  while  asynchronous  delivery  allows  participation  at  differing  times.  The 
main  advantage  of  synchronous  delivery  is  the  provision  for  live  interaction  and  the 
possibility  of  more  natural  group  processes.  The  disadvantage  is  the  requirement  to 
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adhere  to  a  specific  time  frame  that  may  not  be  convenient  for  all  participants,  especially 
those  in  other  time  zones  (Steiner,  1995).  Asynchronous  delivery  systems  are 
characterized  by  the  separation  of  the  instructor  and  student  in  time.  Asynchronous 
systems  allow  anytime-anywhere  learning,  but  are  more  limited  in  student  and  instructor 
interaction. 

As  cited  in  Christensen,  et  al.  (1998),  Alan  Chute  points  out  that  synchronous 
events  are  desirable  in  distributed  learning  programs  in  order  to  provide  student-to- 
student  interaction  for  peer  learning,  and  student-to-instructor  interaction  for  mentored 
learning.  Additionally,  synchronous  events  provide  a  framework  of  calibration  and 
expectations  to  keep  students  on  a  scheduled  track.  Even  programs  that  depend  primarily 
on  asynchronous  learning  benefit  from  periodic  synchronous  events.  Christensen,  et  al. 
(1998)  further  discusses  how  synchronous  learning  activities  leverage  the  great  depth  of 
expertise  and  qualities  of  the  traditional  classroom,  while  expanding  the  physical  reach  to 
geographically  separated  learners. 

The  key  here  is  to  be  able  in  some  way  to  quantify  the  benefits  of  a  synchronous 
training  environment  versus  an  asynchronous  one  in  order  to  determine  which  is  the  more 
appropriate  and  valuable  for  a  particular  set  of  circumstances.  The  following  subsections 
list  several  additional  factors  that  may  contribute  to  potential  returns  on  DL,  which 
should  be  evaluated  when  assessing  the  worth  of  DL  programs.  These  factors  may 
impact  either  positively,  negatively,  or  not  at  all,  but  should  be  considered  nonetheless. 

3.  Benefits  Of  Potential  Returns  from  DL 

TIME  SAVINGS.  With  DL,  the  amount  of  time  students  spend  in  training  can  be 
greatly  reduced.  The  time  previously  spent  in  travel  also  becomes  available  for  work  or 
training.  This  time  “compression”  factor  is  almost  always  a  positive  result  of  course 
conversion  to  a  DL  format  (Belcher,  1999).  A  well-designed  DL  program  requires  less 
student  time  to  achieve  learning  objectives  than  does  traditional  classroom  instruction. 
(An  additional  value-added  result  of  this  process  is  that  the  course  Program  Of  Instruction 
(POI)  is  often  updated  or  modified  during  the  conversion  process,  resulting  in  a  shorter 
overall  course.)  Belcher  (1999)  states  that  asynchronous  learning  allows  students  to  skip 
ahead  if  they  already  know  some  of  the  material.  He  further  suggests  that  because 
students  can  progress  through  these  lessons  at  their  own  pace,  comprehension  may  be 
enhanced.  Some  technologies  allow  students  to  return  to  material  on  the  job  as  refresher 


21 


training.  Studies  conducted  by  the  Institute  for  Defense  Analysis  suggest  that  this 
compression  is  typically  about  30  percent  for  most  forms  of  DL  and  is  occasionally  as 
high  as  60-70  percent  (Metzko,  1996;  Howard,  1997).  For  the  purposes  of  this  study,  the 
more  conservative  value  of  30  percent  will  be  used. 

UNIT  READINESS.  Trainers  argue  that  the  ultimate  objective  of  any  military 
training  or  education  course  is  to  bolster  readiness  (and  subsequently,  performance)  of 
military  units.  The  Army’s  DL  plan  says  as  much  (TADLP,  1999).  DL  technology 
enables  trainers  to  claim  these  readiness  benefits: 

•  Because  some  training  can  be  extracted  from  schoolhouse-based  instruction 
and  done  in  the  operational  units,  recruits  can  spend  less  time  at  initial 
schooling  and  be  assigned  to  operational  units  faster.  They  would  then 
receive  the  additional  training  later  as  required. 

•  Units  can  get  larger  portions  of  their  organizations  trained  on  critical  subjects. 
Since  students  take  less  time  to  learn  some  distance  learning  courses,  they 
spend  less  time  away  from  unit  duties  and  training  than  if  they  took  traditional 
classroom  instruction.  Personnel  are  also  available  in  the  event  of  an 
emergency.  The  result  is  improved  overall  readiness. 

•  The  ability  to  take  training  with  them  (e.g.  on  a  laptop  computer)  enables  units 
to  keep  personnel  even  during  major  field  exercises  and  deployments, 
enabling  the  personnel  to  continue  training. 

ANYTIME,  ANYPLACE  TRAINING.  As  mentioned  before,  one  of  the  ways  for 
DL  to  be  student-centered  is  for  the  student  to  decide  how  learning  takes  place.  The  most 
efficient  DL  programs  will  give  the  student  volitional  control  over  when  and  where  to 
train.  With  minimal  support  equipment,  some  distance  learning  technologies  enable 
trainees  to  access  courseware  at  the  time  and  place  of  their  choosing.  Computer-based 
Instruction  (CBI)  and  CD  ROM-based  instruction  require  only  a  computer  of  modest 
capability.  Internet  access  naturally  extends  that  accessibility  to  Web-based  instruction. 

ENHANCED  COHESION  AND  MORALE.  With  DL,  trainees  spend  more  time 
at  home  stations,  near  friends  and  family,  which  enhances  morale  (Belcher,  1999).  This 
is  particularly  true  for  Navy  shipboard  personnel  who  spend  much  of  their  time  afloat. 
Distributed  courses  thus  allow  for  more  student  and/or  course  flexibility  over 
conventional  schoolhouse  training.  The  advantage  is  tempered  somewhat  from  the  unit’s 
perspective,  however,  by  the  need  to  provide  personnel  duty  time  to  complete  required 
training. 


22 


REDUCED  TRAVEL  COSTS.  The  most  widely  reported  monetary  benefit  of  DL 
is  that  reduced  in-residence  classroom  instruction  minimizes  the  requirement  for  trainees 
to  travel  to  schoolhouses.  Depending  upon  the  size  of  the  training  audience  and  the 
length  of  the  course,  this  can  reduce  travel  and  per  diem  costs.  Travel  and  transportation 
savings  can  be  compounded  when  training  units  use  linked  simulations  or  networked 
simulators.  The  cost  of  transporting  entire  units  and  their  equipment  to  a  central  training 
area  can  be  avoided  (Belcher,  1999). 

INCREASED  EFFICIENCY.  Although  initial  costs  for  many  forms  of  technology 
and  course  conversion  are  high,  they  can  be  offset  by  higher  course  enrollments  over 
time.  The  increased  number  of  people  trained  therefore,  can  thus  recoup  the  start-up 
costs  more  quickly.  An  increasing  portion  of  future  training  infrastructure  will  be  virtual. 
Requirements  for  traditional  schoolhouses  and  training  facilities  will  diminish,  reducing 
overall  infrastructure  costs.  Finally,  the  groundswell  for  Distributed  Learning  is  driving 
the  market  toward  lower-cost  solutions.  As  competition  between  providers  heats  up, 
technology  improves  and  hardware  and  software  costs  will  decline. 

4.  Costs  of  Potential  Returns  from  DL 

The  costs  associated  with  particular  DL  programs/technologies  entail  equal 
consideration  and  include  some  of  the  following  factors: 

COURSE  CONVERSION  COSTS.  Costs  that  should  be  considered  when 
assessing  possible  conversions  are  hardware,  software  and  transmission  conversions,  as 
well  as  time  needed  to  train  faculty  and  staff  (Martin  &  Bramble,  1996).  On  the  plus 
side,  technology-driven  items  such  as  hardware  and  software  items  are  decreasing,  but 
facilities  and  communications  infrastructure  costs  may  apply  for  selected  new  courses. 
Doing  the  work  of  conversion  in-house  can  minimize  course  conversion  costs.  The  two 
principal  considerations  to  lower  course  conversion  costs  are  to  have  the  course  up  to  date 
with  good  presentation  material,  and  to  have  well-developed  standards  and  guidelines  for 
the  courses  (Payne,  1996). 

SUPPORT  COSTS.  For  asynchronous  Web-based  training,  building  and 
maintaining  a  Web  site  requires  planning  for  system  maintenance  and  availability.  If 
students  are  located  around  the  globe,  the  site  should  be  hosted  at  a  location  that  provides 
24  hours  per  day,  7  days  per  week  support.  For  synchronous  training,  technical  support 
personnel  must  be  on-hand  during  sessions.  Instructors  and  students  may  be  working 
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within  narrow  schedules,  resulting  in  a  domino  effect  if  lessons  are  delayed  with  no 
rebroadcast  (Robinson,  1998).  Costs  are  also  associated  with  “training  the  trainer.”  DL 
instructors  must  be  more  organized  than  classroom  instructors.  DL  instructors  require  in- 
depth  knowledge  of  course  material  and  of  the  training  technology.  Some  forms  of  DL, 
like  VTT  can  be  relatively  expensive  to  operate  and  maintain,  but  these  costs  often  are 
overshadowed  by  the  travel  cost  savings  and  the  ability  to  train  larger  numbers  of 
students  simultaneously  (Belcher,  1999). 

STUDENT  EQUIPMENT.  As  DL  becomes  computer  and  Internet  based, 
Internet-capable  student  computers  become  an  absolute  requirement  to  access  instruction. 
Unfortunately,  not  all  students  own  or  have  access  to  these  machines  (Robinson,  1998). 
Training  planners  should  account  for  the  cost  of  equipment  required  by  the  students,  or 
alternatively,  they  must  ensure  that  such  access  is  afforded  to  students.  As  the  equipment 
requirements  change  over  time,  DL  planners  must  determine  how  the  school  or  course 
will  keep  up  with  those  changes. 

5.  Neutral  Factors  that  May  Influence  Returns  from  DL 

Planners  should  further  be  cognizant  of  various  considerations  that  fall  neither  in 
the  benefits  nor  costs  categories.  Many  of  these  issues  surface  as  causes  for  concern  for 
organizations  currently  grappling  with  DL  implementation  issues. 

ACCOUNTING  FOR  INVESTMENTS.  Distance  Learning  conversions  are  often 
the  products  of  various  initiatives  at  various  levels  and  funding  sources.  Organizations 
making  the  investment  in  learning  may  not  necessarily  be  the  same  organizations  reaping 
the  benefits  (ODUSDR,  1999).  Hardware  and  software  supplied  as  government- 
furnished  equipment  (GFE)  to  DL  programs  and  developments  that  result  from 
multiservice  or  govemment/industry  collaboration  further  complicate  the  investment 
chart  of  accounts. 

MATCHING  NEEDS  AND  TECHNOLOGY.  Trainers  must  be  careful  not  to  let 
technology  drive  strategy.  There  is  a  risk  that  plans  will  be  made  and  technologies 
selected  before  the  requisite  analysis  is  performed.  A  careful  match  of  the  training  needs 
and  the  technology  available  should  result  in  cost  savings.  According  to  Sherry  (1996), 
“too  often,  instructional  designers  and  curriculum  developers  have  become  enamored 
with  the  latest  technologies  without  addressing  the  underlying  issues  of  learner 
characteristics  and  needs;  the  influence  of  media  upon  the  instructional  process;  equity  of 


access  to  interactive  delivery  systems;  and  the  new  roles  of  teacher,  site  facilitator  and 
student  in  the  distance  learning  process.”  Student  characteristics  also  must  be  considered 
when  designing  a  course  (i.e.,  independent  work  without  supervision,  number  of  low 
cognitive  tasks,  etc.).  The  student,  not  the  technology,  should  be  the  focus  of  the  design 
(Belcher,  1999). 

COMPLICATED  RESOURCE  SCHEDULING.  Particularly  for  VTT  courses, 
both  transmission  and  downlink  site  availability  must  be  coordinated  for  each  class.  This 
task  can  be  complicated  for  synchronous  training  courses  transmitted  to  remote  locations 
over  several  time  zones.  Such  off-hours  staffing  and  operations  add  to  operations  and 
support  costs  (Belcher,  1999). 

SCHOOLHOUSE  IMPACT.  Traditional  schools  must  digest  a  new  philosophy. 
Training  must  now  follow  a  decentralized,  student-centered  model.  Belcher  (1999)  states 
that  training  should  be  designed  for  how  students  learn,  not  for  how  schools  teach.  To 
him,  flexibility  is  the  key.  In  a  student-centered  model,  the  school  must  be  flexible 
enough  to  respond  to  the  needs  of  the  individual  learner.  Since  DL  courses  yield  higher 
student  throughputs,  they  can  create  backlogs  or  bottlenecks  for  other  related  training 
(Metzko,  1996).  Schools  may  make  portions  of  a  course  available  to  students  via  DL 
technology,  but  they  may  not  produce  more  trained  personnel  because  a  subsequent  phase 
can  only  accommodate  a  few  trainees  (Belcher,  1999).  As  such,  a  more  holistic  view  of 
the  training  and  education  process  must  be  considered  when  implementing  DL  solutions. 
Furthermore,  resource  managers  need  to  consider  new  ways  to  budget  for  schools.  In  the 
past,  military  service  schools  were  allocated  resources  based  on  student  throughput.  In 
the  DL  environment,  this  approach  may  no  longer  be  an  appropriate  measure.  School 
budgets  should  now  focus  on  the  number  and  type  of  courses  the  school  must  convert  and 
execute  (Belcher,  1999).  Finally,  instructor  training/certification  requirements  must  be 
redefined.  The  new  teaching  environment  dictates  that  instructors  be  certified  to  teach 
using  current  DL  technology.  Critical  issues  here  center  on  instructor  competency  with 
hardware/software  components,  as  well  as  flexibility  in  established  teaching  procedures 
and  curricula. 

Belcher’s  (1999)  assessment  is  that  drawing  conclusions  about  ROI  for  DoD  as  an 
enterprise  is  complicated.  Many  initiatives  have  been  undertaken  across  the  services. 
Naturally,  there  has  been  some  redundancy  and  duplication  among  them.  It  is  reasonable 
to  assume  that  eliminating  duplication  across  the  four  services  could  equally  reduce  up¬ 
front  investment  costs  (Metzko,  1996).  The  results,  however,  are  not  purely  additive; 
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determining  just  what  constitutes  duplication  is  subject  to  interpretation.  Case-by-case 
analyses  are  required  to  understand  the  synergies,  opportunities  and  risks  associated  with 
determining  ROI  on  DL. 


C.  COST-BENEFIT  VS.  COST-EFFECTIVENESS  ANALYSIS 

Cost  analysis  can  help  educational  managers  see  the  various  options  and  trade¬ 
offs  available  to  them  and  assess  their  merits  and  feasibility.  It  can  reveal  the  possible 
advantages  of  redeploying  limited  resources  between  different  levels  and  types  of 
education,  among  different  categories  of  inputs,  and  across  different  geographic  areas. 
Cost  analysis  can  uncover  internal  waste  and  recommend  possible  ways  to  eliminate  it.  It 
can  also  suggest  ways  to  enhance  the  external  productivity  of  the  educational  process  and 
the  benefits  accruing  to  individuals  from  investments  in  technology  and  new  processes. 

Cost  analysis  must  be  combined  with  pedagogical  analysis  to  assess  the  outcomes 
of  any  educational  process.  The  key  to  judging  the  efficacy  of  an  educational  process  lies 
in  comparing  the  costs  invested  in  it  and  the  learning  results  attained.  There  is  a 
difference  between  cost-benefit  and  cost-effectiveness.  As  Reynolds  and  Iwinski  (1996) 
state,  cost-benefit  analysis  is  an  attempt  to  weigh  the  costs  of  training  against  the 
outcomes  achieved.  Cost-effectiveness  focuses  on  comparing  the  costs  of  two  or  mere 
training  alternatives,  or  examines  ways  to  reduce  a  program’s  costs,  while  cost-avoidance 
involves  simply  cutting  expenses  to  a  set  level.  This  paper  is  chiefly  concerned  with 
cost-benefit  analysis.  According  to  Gramlich  (1998),  the  fundamental  principle  of  cost- 
benefit  analysis  is  that  when  choosing  among  programs,  the  best  course  of  action  is  to 
choose  the  one  that  maximizes  net  benefits.  To  accomplish  this,  add  up  all  the  gains  from 
a  policy  alternative,  subtract  all  the  losses  and  choose  the  option  that  maximizes  the  total 
benefits. 

Since  life  doesn’t  always  present  simple  alternatives,  however,  there  are  some 
additional  guidelines  to  use  in  applying  this  fundamental  rule.  According  to  Gramlich 
(1998),  one  should  not  compute  cost-benefit  ratios.  He  feels  that  the  right  measure  of  the 
gains  or  losses  to  society  is  program  net  benefits,  and  the  cost-benefit  ratio  can  be  a 
misleading  guide.  His  second  rule  of  thumb  is  that  a  cost-effectiveness  test  can  be  used 
to  get  around  missing  information  when  the  investigator  is  comparing  two  or  more  ways 
of  accomplishing  an  end.  A  third  consideration  is  that  when  projects  are  packaged 
together,  the  net  benefits  test  should  be  applied  to  the  whole  package  (Gramlich,  1998). 
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Cost-benefit  analysis  is  a  framework  for  comparing  the  pros  (benefits)  and  cons  (costs)  of 
project  choices.  Benefits  and  costs  should  be  quantified  whenever  they  can  be  and 
(obviously)  not  when  they  cannot  be;  but  whether  quantified  or  not,  they  should  never  be 
ignored  (Coombs,  et  al.,  1987). 

D.  COST-BENEFIT  TECHNIQUES 


According  to  Reynolds  and  Iwinski  (1996),  there  are  four  considerations 
ordinarily  used  to  analyze  cost-benefits: 

BENEFITS.  Quantifying  benefits  is  a  challenging  task,  because  the  outcomes  of 
training  are  often  intangible  or  difficult  to  translate  directly  into  dollar  amounts.  Factors 
such  as  increased  student  throughput,  reduced  travel  and  per  diem  expenses  and  savings 
in  realia  expenses  (i.e.,  the  cost  of  using  real  things  such  as  ships  or  aircraft)  come  most 
readily  to  mind.  Job  performance  or  organizational  results,  although  difficult  to  quantify, 
oftentimes  more  accurately  justify  a  new  training  program,  and  can  be  expressed  in 
financial  terms  as  well.  Most  technical  training  is  much  easier  to  quantify  than 
management  training  because  it  is  directly  linked  to  processes  and  products  with  known 
or  measurable  value. 

LIFE  CYCLE.  One  of  the  basic  elements  of  CBA  is  life  cycle  cost.  DoD 
Instruction  5000.2  (DoD,  1991)  defines  life  cycle  cost  as  follows: 


Life-cycle  cost  reflects  the  cumulative  costs  of  developing,  procuring, 
operating,  and  supporting  a  system.  They  are  often  estimated  separately  by 
budget  account  (i.e.,  research,  development,  test,  and  evaluation...,  procurement, 
and  operations  and  maintenance).  It  is  imperative  to  identify  life-cycle  costs,  non¬ 
monetary  as  well  as  monetary,  associated  with  each  alternative  being  considered. 

Costs  accrue  over  the  life  of  a  system.  TRADOC  divides  this  life  into  five  distinct  but 
sometimes  overlapping  phases  (Department  of  the  Army,  1985): 

•  Conceptual  (exploratory  development):  Solicitation,  evaluation,  and 
exploration  of  alternative  concepts. 

•  Demonstration  and  Validation  (advanced  development):  Prototypes 
are  produced  to  support  demonstration. 

•  Full  Scale  Development  (engineering):  Prototypes  are  produced  to 
support  operational  test  and  evaluation. 
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•  Production  and  Deployment 

•  Operation  and  Support 

Often  you  can  only  determine  true  cost  savings  by  considering  the  entire  lifetime 
of  the  training  project.  The  life  cycle  model  is  shown  in  Figure  3.1.  The  life  cycle 
method’s  strong  point  is  that  you  can  evaluate  the  total  costs  of  each  of  the  planned 
program’s  phases  to  determine  whether  it  will  result  in  a  net  cost  savings.  Figure  3.1 
illustrates  the  expenditure  profile  for  a  typical  training  project.  Expenditure  corresponds 
to  new  implementations.  The  figures  show  that  some  money  must  be  spent  before 
anything  occurs.  These  costs  are  labeled  as  research  and  development  even  if  no  research 
is  involved.  The  next  phase  is  called  startup  and  marks  a  sharp  rise  in  money  spent  on.the 
training  technology  since  it  has  now  come  of  age.  The  operational  period,  sometimes 
referred  to  as  the  “steady  state,”  is  the  main  period  during  which  the  program  is  active 
(Reynolds  &  Iwinski,  1 996).  This  time  frame  is  much  longer  for  some  technologies  than 
for  others.  Often  a  program  that  performs  its  function  well  and  remains  current,  is 
delivered  by  an  obsolescent  technology  for  years.  Finally,  as  the  program  ages,  it  is 
gradually  phased  out  or  eliminated  during  the  decline  phase.  This  process  is  normally 
accompanied  by  a  simultaneous  transition  to  a  new  program. 

Training  Costs 


STEADY  STATE 


R&D  Start  Up  Operational  Period  Decline  & 

Transition 


Figure  3.1  Technology  Life  Cycle  from  Reynolds  &  Iwinski  (1996) 


PRODUCTIVITY.  This  consideration  compares  both  a  program’s  efficiency  and 
its  effectiveness.  It  may  be  used  to  demonstrate  that  a  project  reduces  training  costs  or 
increases  training.  A  productivity  analysis  can  determine  when  to  switch  from  one 
training  approach  to  another,  or  when  further  application  of  a  particular  approach  will  no 
longer  produce  cost-effective  improvements. 

RESOURCE  REQUIREMENTS.  This  component  is  the  simplest  and  most  direct 
way  to  compare  the  costs  of  two  or  more  different  training  approaches  at  a  given  time 
(Reynolds  &  Iwinski,  1996).  One  merely  determines  the  costs  in  four  major  categories 
for  each  Instructional  Systems  Development  (ISD)  phase  (analysis,  design,  development, 
implementation  and  evaluation)  of  the  particular  training  project.  Make  use  of  the  cost 
categories  equipment,  facilities,  materials  and  personnel.  The  result  of  the  analysis  will 
include  the  total  costs  for  each  phase  and  category.  Resource  requirements  work  well  in 
comparing  a  potential  new  method  with  an  existing  one,  assuming  that  both  applications 
will  be  equally  effective.  To  compare  two  possible  approaches,  one  simply  adds  the  total 
costs  of  each  for  all  phases  and  resource  categories.  Resource  requirements  are  limited  to 
training  costs,  however,  and  should  not  be  used  to  compare  effectiveness.  To  properly 
conduct  a  worthwhile  cost  analysis,  one  must  understand  the  various  factors  that 
comprise  cost  estimation.  The  following  section  will  review  some  simple  fundamentals 
of  cost  analysis  to  include  basic  economic  principles  and  intangible  influences. 

E.  COST  ANALYSIS  IN  TRAINING  DEVELOPMENT  AND  ROI 

1.  Cost  Estimation 

Cost  estimation  should  take  into  account  several  economic  factors.  The  following 
is  distilled  from  a  description  of  key  factors  in  Adams  and  Rayhawk  (1987). 

•  Opportunity  Costs  versus  Accounting  Cost:  Accounting  Cost  is  the 
cost  “on  the  books.”  Opportunity  cost  is  a  hypothetical  value  of  a 
resource  in  it  “best  alternative  use.” 

•  Sunk  Costs:  Costs  that  have  already  been  incurred  and  that  cannot  be 
recouped.  An  example  is  the  cost  of  R&D  spent  on  various  forms  of 
technology. 
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•  Fixed  and  Variable  Costs:  Fixed  costs  are  not  affected  by  how  much 
training  occurs.  An  example  would  be  the  cost  of  classroom  space. 
Variable  costs  vary  with  the  amount  of  training.  An  example  would  be 
the  cost  of  instructors,  whose  number  would  vary  with  the  student 
load. 

•  Time  Value  of  Money:  The  value  of  money  changes  with  time  because 
money  has  earning  power.  This  is  considered  when  comparing 
alternatives  whose  expenses  are  incurred  at  different  rates  over  various 
periods  of  time  by  estimating  both  costs  in  terms  of  “present  value” 
dollars. 

•  Discount  Rates:  Costs  that  can  be  deferred  into  the  future  can  be 
discounted  because  a  smaller  amount  of  money  could  be  invested 
today  and  earn  interest  to  make  the  future  payment. 

•  Constant  versus  Current  Dollars:  The  purchasing  power  (“current 
value”)  of  the  dollar  varies  with  the  general  price  level  and  inflation 
rate.  Constant  dollars  reflect  the  purchasing  power  of  the  dollar  in  a 
selected  base  year. 

•  Residual  Value  of  Assets:  The  value,  if  any,  left  after  a  system  has 
completed  its  life. 

•  Indirect  Benefits:  Benefits  that  may  occur  beyond  the  intended  scope 
of  training.  For  example,  the  value  of  training  to  military  personnel  in 
preparing  them  for  a  civilian  occupation  (Simpson,  1995). 

2.  Common  Cost  Factors 


Reynolds  and  Iwinski  (1996)  state  that  the  most  common  cost  factors  that  affect 
the  development  of  training  programs,  and  impact  on  ROI  include  the  following: 

•  Instructional  strategy.  Technology-based  strategies  cost  more  than 
cramming  students  into  a  large  room  and  talking  at  them,  although 
initial  up  front  investment  costs  may  be  easily  offset  by  increased 
student  throughput. 

•  Existing  course  modification  or  new  course  development.  (New 
courses  cost  more  to  develop,  particularly  in  terms  of  instructor  time.) 

•  Amount  of  existing  information. 

•  Amount  of  individualization. 

•  Type(s)  of  media  used  (Higher  technology  media  cost  more  than 
simpler  technology.) 

These  factors  should  not  be  evaluated  individually,  as  they  often  occur  in  combinations, 
and  thus  affect  the  outcome  of  a  training  program  accordingly. 
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The  cost  of  training  technology  can  be  substantial,  but  perhaps  not  as  severe  as  the 
costs  associated  with  poor  human  performance  within  operational  environments  that 
involve  human  safety  and  well  being.  Mattoon  (1998)  applies  a  disparate  set  of  metrics 
in  his  evaluation  of  ROI  on  training  technologies.  Several  factors  affecting  both  cost  and 
those  less  tangible  benefits  of  effective  training  include  the  following: 

1.  Acceleration.  Does  the  system  accelerate  the  rate  at  which  the  learner 
acquires  knowledge  and  skills? 

2.  Automation.  Does  the  system  support  the  reduction  of  instmctor  workload? 

3.  Availability.  Does  the  system  increase  availability  and  accessibility  of 
training  for  learners? 

4.  Generalizability.  Is  the  system  effective  for  a  variety  of  different  types  of 
learners? 

5.  Longevity.  Does  the  system  contribute  to  greater  retention  of  knowledge  and 
skills? 

6.  Stability.  Does  the  system  exploit  current  technological  capabilities,  emerging 
capabilities,  and  others  that  are  likely  to  come  about  in  the  future? 

7.  Strategics.  Is  the  system  applicable  across  multiple  training  environments? 

8.  Transfer.  Does  the  system  increase  learners’  readiness  to  perform  in  the 
operational  environment? 

Mattoon’ s  approach  to  developing  learner-centered  design  principles  for 
Information  Technology  (IT)  applications  is  based  on  a  synthesis  of  research  on  learning, 
training  and  instruction,  human  factors  engineering  and  human-computer  interface 
design.  His  findings  were  used  to  identify  what  Mattoon  called  “General  Facilitative 
Links”  (GFLs).  This  term  describes  the  relationships  between  technological  capabilities 
and  learning  processes  that  can  be  leveraged  to  improve  knowledge  and  skill 
development.  The  end  result  of  such  leveraging  is  an  increase  on  ROI  for  training.  The 
“General”  component  of  a  GFL  principle  indicates  its  applicability  across  many  different 
learners  and  training  environments.  The  “Facilitative”  term  indicates  the  potential 
positive  impact  a  particular  technological  capability  can  have  on  knowledge  and  skill 
development.  The  “Link”  component  indicates  the  successful  realization  of  this  potential 
through  effective  design  and  implementation  of  Information  Technologies.  Each  of  seven 
specific  GFLs  are  matched  to  the  particular  ROI  factors  discussed  above.  Table  3.1 
illustrates  this  relationship  between  the  GFLs  and  the  ROI  cost  factors. 
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GFL 

ROI  Factors 

1.  Instructional  Simulation 

Combines  instructional  intervention  with  dynamic 
representations  of  conceptual  material  and  criterion 
tasks. 

1,2, 4, 5, 6, 8 

2.  Hyperlinked  Content 

Links  words,  examples,  descriptions,  and  other 
components  of  content  information  together  into  a 
structure  that  depicts  the  type  of  mental  model 
learners  are  intended  to  develop. 

1,2,  3,  6,7 

3.  Distributed  Training 

Employs  network  technologies  to  increase  the 
availability  and  accessibility  of  instruction, 
proficiency  profiles,  and  other  training  materials 
and  tools. 

1,2,  3,  6,7 

4.  Verbal- Visual  Pairing 

Combines  abstract  and  concrete  representations  of 
concepts,  factual  material,  task  components,  and 
relevant  factors  to  facilitate  information  encoding 
and  retention  of  knowledge. 

4,  5,  6,  7 

5.  Multi-modal  Delivery 

Presents  information  in  at  least  two  modes  (e.g. 
audio  dialogue  and  visual  imagery)  to  activate 
selective  attention  and  provide  for  simultaneous 
processing  of  information. 

1,2, 4,  5,  6,7,8 

6.  Proficiency  Tracking 

Monitors  learner  interactions,  captures  data, 
analyzes  performance,  and  summarizes  individual 
progress  and  instructional  needs  within  Proficiency 
Profiles 

1,  2, 4,  6,  7 

7.  Instructional  Control 

Uses  Proficiency  Profile  information  to 
automatically  adapt  instruction,  provide  advice  that 
helps  learners  control  instruction,  or  regulates 
shared  control  between  learner  and  the  IT  program. 

1,  2,  3, 4,  6,  7,  8 

Table  3.1  General  Facilitative  Links  Matched  to  Instructional  Technology  Capabilities  and  ROI 

Factors  from  Mattoon  (1998) 


There  are  many  potential  GFLs  that  can  be  identified  to  help  promote  the  design 
of  more  effective  training.  Because  GFLs  represent  general  principles,  continued 
identification  of  new  GFLs  is  likely  to  continue  and  lead  to  one  of  two  outcomes:  (a)  the 
range  of  applicability  of  GFLs  will  begin  to  overlap  to  the  extent  that  individual 
distinction  among  them  becomes  academic  rather  than  practical;  or  (b)  the  degree  of 
generality  or  breadth  of  application  of  each  GFL  will  become  more  specific.  According 
to  Mattoon  (1998),  the  latter  outcome  is  more  preferable,  because  it  leads  to  the 
identification  and  application  of  “Specific  Facilitative  Links”  (SFLs).  SFLs  would 
provide  more  detailed  and  clearly  defined  boundaries  for  application  to  specific  training 
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curriculum  and  challenges.  The  same  logic  holds  for  the  identified  ROI  Factors  that  may 
be  constrained  to  specific  priorities,  cost  drivers  and  goals  of  a  particular  training 
operation.  This  degree  of  specificity  is  necessary  before  cost-benefit  analysis  can  be 
effectively  applied  to  training  technologies  to  accurately  predict  ROI  (Mattoon,  1998) 

F.  MEASURES  OF  EFFECTIVENESS  FOR  ROI 

In  any  orderly  process  designed  to  provide  a  solution  to  a  stated  need,  there 
should  always  be  a  provision  for  answering  the  question  “How  do  I  know  if  I  have 
succeeded?”  The  way  to  do  this  is  to  establish  measures  against  which  the  solution(s) 
will  be  tested  and  evaluated  (Sproles,  1997).  When  used  in  conjunction  with  test  and 
evaluation,  measures  are  units  for  the  quantification  of  qualities  of  the  system  or  entity 
under  review.  Based  on  extensive  research  conducted  by  Donald  Kirkpatrick  (1994),  the 
U.S.  Navy  has  applied  their  own  version  of  Measures  of  Effectiveness  (MOE)  to  training 
technologies  to  better  gauge  their  probability  of  success.  According  to  a  draft  CNET 
Instruction  on  ROI  for  training  program  investments  (CNET  INSTRUCTION  3920. 1 J,  1 
April  96),  the  U.S.  Navy  recognizes  four  different  levels  of  ROI  analysis  that  can  be 
conducted  when  evaluating  a  potential  program.  Student  and  instructor  reaction  is  the 
first  level  of  analysis  and  is  also  the  most  commonly  used  method  of  evaluating  training 
effectiveness;  so  much  so,  that  it  is  considered  a  default  MOE.  Student  learning 
achievement  is  the  second  level  of  training  effectiveness  evaluation.  Student  learning 
achievement  is  the  most  important  quality  indicator;  however,  student  learning  is 
influenced  by  a  large  number  of  external  variables  that  are  viewed  as  outside  the  trainer’s 
control  (e.g.,  attitude,  motivation,  aptitude  and  transfer  environment).  Consequently, 
measures  of  learning  are  most  reliable  when  they  are  taken  inside  the  system  in  which  the 
trainer  does  exercise  control  (e.g.,  in  formal  schools  versus  in  the  fleet).  Evaluations  of 
student  performance  in  the  fleet  thus  encompass  the  third  level  of  training  effectiveness 
evaluation.  It  should  be  noted  that  this  type  of  performance  analysis  is  extremely 
expensive  to  conduct  correctly,  as  it  attempts  to  control  for  variables  that  in  many  cases 
have  nothing  to  do  with  training  quality  in  the  schoolhouse,  and  is  therefore  considered 
highly  unreliable.  These  first  three  levels  together  make  up  what  is  termed  the  quality 
improvement  measures  of  effectiveness.  Specific  factors  that  comprise  this  MOE  include 
the  following: 
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1 .  Quality  Improvement  Measures  Of  Effectiveness 

a.  Final  course  test  scores 

b.  Follow-on  course  test  scores 

c.  Reductions  in  academic  review  boards 

d.  Fleet  returnee  feedback  (Subjective,  Qualitative) 

e.  Student  course  critiques  (Subjective,  Qualitative) 

The  next  level  of  analysis  forms  the  core  of  the  CNET  Training  Technology  ROI 
procedures.  These  measures  address  the  business  results  and  financial  impact  of  training 
technology  investments.  According  to  the  instruction,  progress  against  the  upcoming 
Program  Objective  Memorandum  (POM)  Individual’s  Account  reduction  goals  are 
measured  against  timesaving  measures  of  effectiveness.  Direct  cost  savings  are  measured 
using  cost  savings  measures  of  effectiveness. 

2.  Time  Savings  Measures  Of  Effectiveness 

a.  Reduced  Individual’s  Account  expenditures 

(1)  Implemented  reductions  in  course  length 

(2)  Reductions  in  average  on  board  (AOB)  resulting  from: 

(a)  Reductions  in  students  awaiting  training 

(b)  Reductions  in  students  under  instruction  time 

(c)  Reductions  in  interrupted  instruction  time 

(d)  Reductions  in  students  awaiting  transfer  time 

b.  Reductions  in  attrition 

c.  Reductions  in  setbacks 

d.  Reduced  TAD/TDY  transit  time 

e.  Reductions  in  instructor  personalization  and  course  revision  time 

3.  Cost  Savings  Measures  of  Effectiveness 

a.  Reductions  in  training  consumables 

b.  Reductions  in  overall  TAD/TDY/PCS  costs 

Based  on  these  direct  and  simplified  MOEs,  the  Navy’s  intent  is  to  provide  an  easily 
understandable,  systematic  and  consistent  management  tool  whereby  the  Chief  of  Naval 
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Operations  (CNO)  and  CNET  can  measure  the  efficiency  of  training  technology 
investments  and  thus  compare  progress  against  predetermined  budget  reduction  goals. 

G.  CHAPTER  SUMMARY 

Distance  learning  is  not  just  about  technology.  It  is  not  a  singular  strategy,  but  is 
a  part  of  an  integrated  strategy  for  training.  The  learner,  not  the  technology  should  be 
the  focus  of  effort.  This  concept  is  fundamental,  so  when  assessing  returns  one  should 
concentrate  less  on  the  costs  to  the  trainer  and  more  on  the  benefits  to  the  trainee.  As 
previously  mentioned,  however,  these  benefits  may  sometimes  be  difficult  to  accurately 
measure. 

Return  On  Investment  for  a  military  service  course  is  especially  difficult  to  define 
and  compute  for  a  number  of  reasons.  Many  relevant  factors  are  not  easily  quantifiable  in 
monetary  terms.  Moreover,  returns  are  not  always  clearly  stated  in  monetary  terms.  The 
one  benefit  touted  most  often  is  that  DL  saves  student  transportation  and  per  diem  costs. 
The  most  interesting  data  found  by  Belcher  (1999),  however,  was  not  that  DL  courses 
save  money  (although  most  do),  but  that  they  provide  other  benefits.  These  other  benefits 
range  from  time  saved  to  improved  training.  Factors  like  time-to-train,  flexibility  of 
usage  and  impact  on  unit  readiness  should  weigh  heavily  in  a  student-centered 
assessment. 

Drawing  “before  and  after”  conclusions  about  the  introduction  of  DL  into  a 
curriculum  can  prove  complicated  because  costs  directly  related  to  converting  a  course  to 
a  DL  format  are  difficult  to  isolate.  Reasons  for  this  include  the  fact  that  many 
organizations  that  convert  courses  to  DL  formats  often  use  the  same  opportunity  to 
update  or  change  the  course’s  program  of  instruction  (POI).  The  resultant  course  may  be 
completely  different  from  the  original  course.  If  these  course  changes  were  predictable 
and  consistent,  then  they  could  be  incorporated  into  ROI  estimates  a  priori. 
Unfortunately,  they  are  most  often  not,  so  estimating  the  final  ROI  traceable  to  the 
technology  insertion  is  difficult.  Thus  many  course  conversion  costs  and  funding  sources 
are  hard  to  isolate.  Course  modification  often  is  coincident  with  DL  conversion,  while 
some  conversions  are  actually  new  courses  or  only  partial  conversions.  Courses  may  not 
be  completely  converted.  In  some  instances,  schools  have  chosen  to  convert  only 
portions  of  a  POI  to  DL  format.  They  believe  it  is  necessary  to  maintain  some  of  the 
classroom  instruction  because  of  the  nature  of  the  particular  training  (Belcher,  1999). 
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Some  courses  are  designed  from  scratch  as  DL  courses.  In  those  instances,  of  course, 
there  is  no  training  “before”  state  with  which  to  compare  the  benefits  of  the  new  course 
(at  least,  none  that  would  not  be  realized,  at  least  partially,  by  instituting  a  classroom- 
based  course).  The  components  of  a  DL  conversion  may  be  the  products  of  multiple 
contributory  resources  or  from  different  pots  of  money.  For  instance,  an  Army  VTT 
course  may  be  the  beneficiary  of  proponent  school  courseware  funding,  a  training  support 
center  paper,  or  CD-ROM  read  ahead  materials.  Additional  competing  sponsors/users 
may  include  the  Training  and  Doctrine  Command,  the  Army  Distance  Learning  Program 
Manager,  National  Guard-funded  VTT  classrooms,  or  even  DoD-funded  satellite  time 
each  with  their  own  agenda  (Belcher,  1999).  Finally,  much  data  relevant  to  determining 
ROI  are  not  collected.  Success  stories  often  collect  data  on  only  one  aspect  of  training 
(e.g.,  reduction  in  student  training  hours)  without  tracking  the  cost  of  development,  cost 
of  fielding,  cost  of  operations  and  maintenance  or  student  test  results. 

To  provide  the  reader  with  a  better  perspective  and  understanding  of  the  numerous 
factors  pertinent  to  cost  analysis  of  training  and  education  programs,  this  chapter  focused 
on  specific  issues  that  have  been  found  to  impact  on  ROI  for  DL  courses.  Information  is 
provided  briefly  outlining  the  differences  and  appropriateness  of  CBA  versus  CEA  in 
evaluating  alternatives,  followed  by  a  discussion  of  suitable  cost-benefit  analysis 
techniques  and  common  terminology.  Although  some  studies  conclude  that  there  is  no 
significant  difference  in  the  ultimate  learning  effectiveness  between  classroom  instruction 
and  Advanced  Distributed  Learning-enhanced  instruction  (Russell,  1999),  the  issue  is  not 
a  foregone  conclusion.  Much  debate  continues  over  this  contentious  point,  particularly  in 
regard  to  the  MOEs  that  may  be  used  in  an  analysis.  In  spite  of  this  disagreement, 
however,  there  are  clearly  identified  increases  in  efficiency  that  if  applicable,  should  be 
included  in  any  assessment  of  DL  programs.  Accordingly  a  list  of  MOEs  commonly  used 
by  the  U.S.  Navy  for  training  and  education  programs  was  included  as  well. 
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IV.  PRELIMINARY  OBSERVATIONS 


A.  METHODOLOGY 

To  gain  a  perspective  of  how  DL  initiatives  and  Interactive  Multimedia 
Instruction  (IMI)  technologies  have  actually  been  applied  to  date  by  the  USMC,  I  chose 
to  review  the  curricula  of  selected  courses  currently  taught  at  the  Marine  Corps 
Communications  and  Electronics  School  (MCCES),  located  at  the  Marine  Corps  Air 
Ground  Combat  Center  (MCAGCC),  29  Palms,  California.  My  purpose  for  choosing  this 
particular  location  was  based  on  the  following  criteria: 

(1)  MCCES  presently  has  one  of  the  most  advanced,  state  of  the  art  training 
facilities  in  terms  of  implemented  training  technologies,  as  outlined  in  the 
USMC’s  Strategic  Roadmap  for  DL  and  CBT. 

(2)  The  variety  of  courses  taught  at  MCCES  are  among  the  most  lengthy  and 
challenging  within  the  USMC,  ranging  from  several  weeks  to  over  a  year  in 
duration. 

(3)  The  quality  of  the  instruction  provides  skills  that  are  in  high  demand  in  both 
the  USMC  and  civilian  industry.  Turnover  in  these  particular  technical  MOSs 
is  frequent,  resulting  in  relatively  high  and  consistent  student  throughput. 

(4)  While  functioning  as  an  FLC,  MCCES  has  been  further  designated  as  one  of 
only  two  RDCs  that  will  support  all  training  development  for  the  West  Coast. 
Moreover,  with  the  installation  of  a  TEPOP  server,  MCAGCC  serves  as  an 
ALC,  and  additionally  has  a  fully  functional  VTT  facility.  In  short,  all  the 
critical  components  of  the  USMC’s  DL  program  are  in  place  at  this  single 
location. 

As  previously  discussed,  the  four  categories  of  learners  found  within  the  USMC’s 
organizational  hierarchy  consist  of  the  following: 

(1)  Initial  skills  training,  to  include  recmit  training,  Marine  Combat  Training,  and 
MOS  school. 

(2)  Advanced  skills  training. 

(3)  Professional  Military  Education. 

(4)  Marines  Awaiting  Training. 


For  the  purpose  of  this  paper,  I  chose  to  limit  my  research  to  categories  (1),  (2) 
and  (4).  Debate  continues  over  the  applicability  of  DL  initiatives  to  entry  level  skills 
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training.  I  therefore  conducted  my  analysis  of  two  training  companies  within  MCCES; 
Company  A,  which  trains  technicians  and  maintenance  personnel,  and  Company  B, 
which  trains  equipment  operators.  Within  each  of  the  companies,  I  further  focused  my 
research  on  an  entry-level  course  and  an  advanced  skills  level  course,  looking  for  specific 
problem  areas. 

The  courses  reviewed  in  Company  A  were  the  Basic  Electronic  Repair  Course,  an 
introductory  level  course  designed  to  provide  students  with  elementary  concepts  of 
electricity,  electronics,  digital  logic,  computer  operation  and  basic  electronic  construction 
techniques.  This  initial  instruction  provides  the  foundation  for  further  training  in  the 
maintenance  of  telecommunications  or  electronics  equipment  and  eventual  qualification 
in  MOSs  2800  or  5900.  This  course  must  first  be  successfully  completed  before  the 
student  can  progress  to  the  next  phase  of  training.  I  next  looked  at  the  Technician  Theory 
Course,  which  consists  of  advanced  level  training  for  senior  Marines,  in  which  students 
are  introduced  to  sophisticated  aspects  of  transmission  theory  and  applications  and 
provided  instruction  in  advanced  digital  technology. 

Following  a  similar  approach  in  Company  B,  I  reviewed  the  Basic  Field 
Wireman’s  Course,  which  provides  entry-level  instruction  in  the  operation  of  analog  and 
digital  wire  communications  equipment.  I  then  looked  at  the  Communications  Systems 
Chief  Course  (CSCC)  and  the  Operations  Communications  Chief  Course  (OCCC).  The 
CSCC  provides  advanced  level  instruction  in  the  three  primary  communications  systems 
that  the  USMC  currently  uses;  radio,  wire  and  data  networks;  while  the  OCCC  trains 
students  in  communications  planning,  including  information  on  the  organization  and 
employment  of  USMC  command  and  control  systems,  as  well  as  routine  communications 
support. 

I  looked  at  such  factors  as  total  student  throughput,  failure  and  remediation  rates, 
changes  in  interactive-engagement  methods,  student-to-instructor  ratios,  reductions  in 
overall  course  hours  and  the  potential  for  DL  applications  for  each  course  both  before  and 
after  the  inclusion  of  CBT  and  IMI  technologies.  To  gain  a  better  understanding  of  these 
issues,  I  interviewed  over  two  dozen  instructors  and  administrative  personnel,  officers, 
civilians  and  enlisted  alike,  from  both  of  the  training  companies,  as  well  as  support 
personnel  from  the  MCCES  staff.  I  also  questioned  the  system  administrators  who 
supervised  the  LRC  at  MCAGCC.  The  experience  levels  of  the  interviewees  varied,  with 
some  individuals  having  been  present  at  the  schoolhouse  for  a  couple  of  years,  both 
before  and  after  the  implementation  of  CBT  in  the  classroom,  while  others  had  only  just 
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recently  arrived.  The  variance  in  experience  levels,  combined  with  the  degree  of 
seniority  and  knowledge  of  the  personnel  involved,  provided  an  accurate  representation 
of  the  differing  opinions  regarding  the  inclusion  of  CBT  in  the  classroom,  allowing  me  to 
accurately  capture  many  of  the  concerns  surrounding  training  technology  issues.  I  further 
reviewed  the  problem  of  Marines  Awaiting  Training,  focusing  on  how  information 
technologies,  to  include  DL,  have  addressed  this  issue.  Finally,  I  examined  the  use  of  the 
LRCs,  both  at  the  schoolhouse  and  the  base,  to  identify  any  trends  in  the  levels  of  use 
since  they  were  opened. 

B.  OVERVIEW  OF  MCCES 

The  training  constraints  currently  faced  by  MCCES  originate  from  the  guidance 
provided  by  former  Commandant,  General  Charles  Krulak  (Krulak,  1995),  for  all  formal 
schools  to  reduce  the  time  Marines  spend  in  entry  level  training,  in  order  to  get  them  to 
operational  units  faster,  thus  reducing  the  overall  number  of  Marines  in  the  T2P2  account. 
The  more  time  junior  Marines  languish  in  a  training  status,  the  less  return  the  Marine 
Corps  can  expect  from  its  recruiting  investment. 

Based  on  the  personal  anecdotes  of  instructors  and  administrators,  the  planned 
reductions  in  individual  course  length,  for  example,  were  not  carefully  considered,  in 
many  cases  resulting  in  arbitrary  reductions  in  various  courses.  Procedures  for  formal 
course  revisions,  as  outlined  in  the  Systems  Approach  to  Training  (SAT)  manual,  were 
ignored.  Input  from  Subject  Matter  Experts  (SMEs)  regarding  the  retention  or 
elimination  of  Individual  Training  Standards  (ITSs)  from  a  particular  course  of 
instruction,  and  normally  reviewed  during  a  Course  Curriculum  Review  Board  (CCRB), 
was  not  requested.  In  some  cases,  the  training  companies  were  directed  to  reduce  overall 
course  lengths  by  an  across-the-board  percentage  without  regard  for  what  would  be 
eliminated.  It  was  felt  that  new  CBT  and  DL  technologies  would  increase  the  overall 
efficiency  of  the  learning  process,  as  conventional  classroom  time  was  reduced.  As  a 
consequence,  however,  some  courses  were  cut  too  deeply  with  reductions  as  great  as  50 
percent  of  critical  ITSs  completely  removed.  When  asked  for  their  opinion  on  whether 
CBT  has  made  up  for  the  reductions  in  conventional  classroom  time,  responses  were 
mixed.  Senior  instructors  felt  that  CBT,  although  useful,  was  not  a  substitute  for  the 
mandated  reductions  in  conventional  classroom  time.  Junior  instructors  favored  CBT  and 
felt  that  it  improved  overall  instruction  in  terms  of  learning  transfer  and  retention.  These 
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junior  instructors  had  been  teaching  prior  to  the  activation  of  the  Automated  Electronic 
Classroom  (AEC),  and  had  the  benefit  of  conventional  classroom  experience. 

Instructors  reported  that  frequent  complaints  were  received  by  MCCES  from 
operational  units  regarding  the  training  of  junior  Marines  arriving  from  the  schoolhouse 
since  the  activation  of  the  AECs.  Taking  into  the  account  the  experience  level  of  senior 
non-commissioned  officers  in  the  operational  units,  senior  instructors  and  administrators 
from  the  schoolhouse  determined  that  most  of  the  complaints  were  unfounded  and  based 
mainly  on  conjecture,  as  the  experience  levels  of  junior  Marines  arriving  prior  to  the  use 
of  CBT  were  never  evaluated.  No  baseline  therefore  existed  against  which  to  make  such 
claims. 

The  current  period  under  observation  (Winter/Spring)  was  considered  the  busiest 
time  of  the  year  in  terms  of  student  throughput  for  the  schoolhouse.  Larger  numbers  of 
Marines  recruited  during  the  summer  months  (after  graduation  from  high  school)  having 
completed  recruit  training  and  Marine  Combat  Training  (MCT),  were  now  receiving 
entry  level  MOS  training.  In  accordance  with  the  TEMI,  the  fielding  plan  for  the 
acquisition,  installation  and  operation  of  an  integrated  telecommunications  system  for 
distribution  of  IMI  and  training  software  was  on  schedule.  The  hardware  and  software 
components  of  MCCES  and  MCAGCC  are  currently  in  place  and  fully  operational. 
Accompanying  changes  in  the  traditional  methods  of  instruction  to  include  course  length, 
curriculum  content  and  delivery  mechanisms  had  only  recently  been  completed  and 
implemented  the  preceding  summer.  Little  had  been  done  prior  to  my  visit  to  implement 
formal  DL  solutions  to  improve  instructional  effectiveness.  There  was  unanimous 
agreement  that  DL  technologies  should  not  be  applied  to  entry  level  skills  training.  They 
should  be  used  for  advanced  skills  ( core  plus)  level  courses. 

C.  COMPANY  A 

Company  A  provides  training  for  technicians  and  maintenance  personnel.  The 
entry-level  course  reviewed  was  the  Basic  Electronic  Repair  Course  (Course  ID# 
M092721).  Course  length  is  approximately  40  training  days.  Class  capacity  ranges  from 
a  minimum  of  15  students  to  a  maximum  of  30  students.  There  are  50  classes  held  per 
year  with  an  average  annual  throughput  of  1300  Marines.  This  figure  is  in  accordance 
with  the  Training  Input  Plan  (TIP),  which  estimates  an  annual  throughput  of 
approximately  1325  Marines  for  FY01.  It  should  be  remembered  that  this  is  a  feeder 
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course  for  further  entry  level  training,  so  depending  upon  the  individual  Marine’s 
particular  MOS  (either  2800  or  5900  field),  additional  training  time  may  last  over  a  year 
(Herring,  2000). 

1.  LAB  VOLT  Experiment 

This  course  was  of  particular  interest  to  my  study  due  to  the  use  of  some 
innovative  automated  electronic  training  technologies  that  had  been  recently  instituted. 
Known  as  LAB  VOLT,  this  technical  training  system  features  educationally  substantive, 
competency-based  curriculum  providing  hands-on  activities  for  learning,  testing, 
troubleshooting,  applying  and  designing  basic-to-advanced  electronic  circuits.  The 
system  consists  of  a  breadboard  of  basic  electronic  circuits  connected  to  an  interactive 
computer  program  designed  to  walk  the  student  through  each  lesson.  The  system  offers 
stand-alone  capability  in  which  the  student  can  progress  at  their  own  pace,  as  well  as  a 
test  and  remediation  package  that  guides  the  student  through  each  progressive  skill  level. 
Based  on  discussions  with  course  instructors,  this  particular  system  was  designed  for 
civilian  schools  and  has  been  used  with  success  by  the  ITT  Technical  Institute  and  DeVry 
Institute  of  Technology.  It  was  pointed  out  that  many  of  these  institutions  furnished  only 
one  computer  for  two  to  three  students.  This  was  felt  to  be  inadequate  for  this  type  of 
instruction.  Accordingly,  the  number  of  students  per  class  for  the  Basic  Electronic  Repair 
Course  was  capped  at  27  to  ensure  a  terminal-to-student  ratio  of  1 :1  (Weber,  2000). 

A  study  at  the  Marine  Corps  Training  and  Evaluation  Branch  revealed  that  since 
the  application  of  this  technology,  the  overall  course  length  was  reduced  from  55  to  40 
days,  but  most  interestingly,  the  overall  recycle/failure  rate  was  reduced  from 
approximately  46  percent  to  6  percent.  Upon  closer  inquiry  with  instructors  and  staff,  it 
was  revealed  that  the  results  were  even  more  impressive.  Under  the  traditional  lecture 
method,  approximately  80  percent  of  the  students  were  set  back  at  least  once  during  the 
11-week  course.  After  the  full  implementation  of  the  LABVOLT  curriculum,  setbacks 
were  reduced  to  15  percent.  The  attrition  rate  remained  about  the  same.  As  impressive  as 
these  results  were,  however,  it  was  soon  discovered  that  although  the  system  was 
designed  to  stand  alone,  after-action  comments  from  students  indicated  that  difficulties 
were  still  being  encountered  in  terms  of  learning  transfer  and  retention.  As  indicated  in 
the  Program  of  Instruction  (POI)  for  this  course,  the  number  of  instructors  per  class  was 
two.  Apparently,  while  the  instructors  were  busy  answering  questions  and  helping  the 
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more  vocal  students  experiencing  difficulty,  others  would  grow  impatient  and  simply 
press  on  through  the  program  without  waiting  to  have  their  questions  answered.  This 
problem  came  to  light  when  instructors  later  went  back  and  checked  the  progress  of  the 
students.  The  program  was  designed  in  such  a  way  so  as  to  see  exactly  where  each 
individual  student  experienced  difficulty,  if  any,  by  the  way  in  which  the  questions  were 
answered  (DeLeon,  2000). 

As  a  result  of  these  initial  experiences  with  the  LABVOLT  system,  it  was  decided 
to  reintroduce  lectures  back  into  the  POI  to  aid  in  retention  of  basic  information.  This 
action  not  only  increased  retention,  but  also  actually  allowed  the  students  to  progress 
through  the  lessons  at  a  faster  rate,  so  no  time  was  lost  in  the  overall  schedule.  The 
instructors  felt  that  the  use  of  such  lectures  prior  to  the  start  of  each  period  of  instruction 
oriented  the  students  and  enabled  them  to  better  understand  and  comprehend  key 
concepts  and  ideas  as  they  continued  independently.  It  was  also  found  that  the  overall 
number  of  questions  significantly  decreased  when  the  lectures  were  reintroduced  (Weber, 
2000). 


2.  Marines  Awaiting  Training 

Another  area  of  interest  concerned  how  Company  A  had  addressed  the  issue  of 
Marines  Awaiting  Training  (MAT).  Through  the  use  of  such  technology  as  outlined 
above,  the  Company  was  able  to  reduce  its  MAT  status  from  an  average  of  four  months 
to  an  average  of  30  days  (An  overall  reduction  of  75  percent).  Much  of  this  reduction 
was  attributed  to  the  faster  progression  rates  the  students  experienced  as  a  result  of  the 
LABVOLT  system.  Upon  closer  examination,  however,  it  was  found  that  much  of  this 
reduction  in  MAT  was  achieved  by  conducting  the  course  24  hours/day,  in  shifts,  with 
half  the  instructors  providing  instruction  during  the  day,  and  the  other  half  teaching  at 
night.  Although  successful  in  significantly  reducing  the  numbers  of  Marines  in  MAT 
status,  the  problems  encountered  included  such  issues  as  equipment  breakdown  from 
overuse  and  lack  of  scheduled  maintenance;  instructor  burnout  and  exhaustion;  and  most 
importantly,  rather  than  being  eliminated,  the  MAT  issue  was  only  shifted  to  a  follow-on 
phase  of  the  Marine’s  training,  as  the  other  courses  could  not  handle  such  large 
throughput  at  the  same  rate.  As  a  result,  the  number  of  courses  per  year  was  reduced  to 
current  levels  (DeLeon,  2000). 
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An  alternative  approach  to  this  problem  used  by  Company  A  was  to  provide  the 
Marines  in  MAT  with  follow-on  training  that  had  previously  taken  place  within  the  first 
six  months  that  the  Marines  arrived  at  their  operational  units.  Training  included  events 
such  as  uniform  refitting,  gear  accountability  and  inspections,  physical  fitness,  drill,  core 
value  instruction,  sexual  harassment  training,  etc.  Not  only  does  such  activity  keep  the 
Marines  suitably  occupied,  but  also  allows  the  Marine  to  be  put  directly  to  work  upon 
reaching  his  permanent  duty  station.  This  approach  has  received  favorable  endorsement 
and  feedback  from  operational  units.  Unfortunately,  this  technique  was  not  in  practice 
throughout  MCCES,  as  each  training  company  exercises  its  own  method  of  handling  the 
MAT  issue.  Current  figures  for  the  Basic  Electronic  Repair  Course  at  the  time  of  this 
research  consisted  of  434  Marines  currently  in  formal  training,  152  in  MAT  status 
(informal  training),  and  10  personnel-awaiting  disposition  (i.e.,  possible 
failures/recycles).  Obviously,  the  issue  of  MAT  was  not  a  problem  for  the  advanced 
skills  level  courses  due  to  smaller  student  throughput  and  scheduling  practices. 

3.  Technician  Theory  Course 

The  Technician  Theory  Course  (TTC)  (Course  ID  #  M09TA31)  is  approximately 
80  training  days  in  length.  As  mentioned,  it  is  an  advanced  skill  level  training  course 
designed  to  provide  instruction  to  Marines  in  the  grade  of  Corporal  through  Gunnery 
Sergeant  on  a  second  or  subsequent  enlistment.  According  to  the  Course  Descriptive 
Data  (CDD)  sheet,  student  capacity  ranges  from  a  minimum  of  12  students  to  a  maximum 
of  24  students.  There  are  10  classes  held  per  year  with  a  total  student  throughput  of  188 
Marines  per  year,  which  is  in  accordance  with  the  TEP.  Each  individual  class  averages  24 
Marines.  This  course  has  also  been  fully  converted  for  use  in  an  Automated  Electronic 
Classroom  (AEC)  environment,  although  the  specific  lessons  are  not  fully  interactive  as 
in  the  LABVOLT  example.  As  an  advanced  skill  level  course,  providing  core-plus 
training,  DL  applications  were  viewed  favorably  by  instructors  and  administrators  alike. 
Based  on  interviews  with  schoolhouse  personnel,  it  was  determined  that  as  much  as  50 
percent  of  the  current  curriculum  could  be  disseminated  via  DL  technologies,  thus 
reducing  overall  course  length  significantly  (Herring,  2000).  Not  only  would  prospective 
students  be  able  to  review  and  validate  much  of  the  introductory  information,  they  would 
begin  the  course  on  a  relatively  level  playing  field  in  terms  of  knowledge  and  thus  be  able 
to  absorb  follow-on  material  much  more  quickly. 
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A  word  of  caution  is  in  order  here,  however,  regarding  course  length  reductions 
based  on  predicted  savings  from  DL  applications  and  other  automated  training 
technologies.  One  of  the  greatest  frustrations  expressed  by  instructors  and  administrators 
alike  was  the  inclination  of  HQMC  to  mandate  reductions  in  overall  course  lengths  based 
on  predicted  savings  from  advanced  training  technologies.  Particularly  in  regard  to  the 
entry-level  courses,  it  was  felt  that  time  saved  by  the  introduction  of  automation  into  the 
classroom  should  be  used  to  provide  additional  training  rather  than  eliminated  from  the 
curriculum.  The  reasoning  was  that  junior  Marines  require  as  much  exposure  as  possible, 
to  the  numerous  systems  currently  entering  the  USMC’s  inventory  that  they  will  be 
required  to  operate  and  maintain.  Under  the  current  guidance  to  reduce  training  to  the 
absolute  minimum  and  get  the  junior  Marine  to  the  fleet  as  soon  as  possible,  this  is  not 
taking  place.  Hence,  the  complaints  from  operational  units  that  the  overall  quality  of  new 
Marines  arriving  from  MCCES  has  dropped.  This  argument  was  not  found  to  carry  equal 
weight  in  regard  to  advanced  skills  training,  however,  as  the  key  with  more  senior, 
experienced  Marines,  is  to  keep  them  in  the  operational  units  as  much  as  possible  where 
their  expertise  is  needed.  Advanced  skill  level  instruction  is  a  training  niche  for  which 
DL  seems  exceptionally  well  suited. 

Accordingly,  measures  are  currently  being  developed  to  adopt  this  course  to  a  DL 
format,  although  no  specific  date  was  set  for  this  to  occur.  The  predicted  savings  from 
such  action  were  expected  to  be  significant,  based  on  the  mythical  training  day  cost  of 
$271  per  Marine.  I  enquired  into  the  source  of  this  $271  figure,  seeking  to  determine 
where  it  came  from,  and  upon  what  assumptions  it  was  based.  No  one  within  Company 
A  or  MCCES  could  explain  where  it  originated,  or  upon  what  assumptions  it  is  based. 
Nonetheless,  it  is  used  frequently  for  budgeting  purposes.  It  would  therefore  appear  that 
although  the  estimated  savings  from  the  introduction  of  DL  technology  into  this 
particular  course  may  be  substantial,  the  actual  value  will  depend  upon  a  determination  of 
what  it  actually  costs  to  train  a  Marine  per  day  for  this  particular  course.  In  all  fairness  to 
MCCES,  however,  it  should  be  noted  that  using  antiquated  or  inaccurate  costing  data  is  a 
convenient  practice  that  occurs  throughout  military  and  civilian  budgeting  (Gramlich, 
1998). 
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D.  COMPANY  B 


Company  B  provides  both  entry-level  and  advanced  skill  training  for 
communications  systems  operators.  The  Basic  Field  Wireman  Course  (Course  ID# 
M092471),  which  was  examined  as  an  entry-level  program,  specifically  furnishes 
instruction  in  the  operation  of  Fleet  Marine  Force  analog  and  digital  wire 
communications  equipment.  It  encompasses  the  application,  installation,  adjustment  and 
familiarity  with  the  operational  characteristics  and  employment  of  wire/telephone 
equipment.  Additional  requirements  of  this  course  include  demonstrated  proficiency  in 
communication  procedures,  and  basic  knowledge  of  relevant  publications,  directives  and 
security  regulations.  Course  length  is  approximately  17  training  days.  Class  capacity 
ranges  from  a  minimum  of  20  students  to  a  maximum  of  40.  There  are  16  classes 
convened  per  year  with  an  average  annual  throughput  of  640  Marines,  which  in  this 
instance  was  in  excess  of  the  TIP  requirement  of  596.  The  discrepancy  can  be  attributed 
to  the  fact  that  the  TIP  has  not  yet  been  updated  for  this  particular  course,  which  was 
reduced  in  length  from  40  to  17  training  days. 

1.  Basic  Field  Wireman’s  Course 

This  reduction  in  overall  course  length  represented  perhaps  the  most  extreme 
example  that  was  found  regarding  the  application  of  CBT  in  the  traditional  classroom 
setting.  As  such,  there  was  also  the  most  diverse  range  of  opinions  regarding  the  success 
of  this  action.  There  was  general  agreement  among  all  the  instructors,  both  young  and 
old,  that  the  course  had  indeed  been  cut  too  severely,  leaving  no  time  to  teach  many  of 
the  critical  tasks  and  skills  needed  by  the  Marines  to  successfully  function  in  their 
operational  billets.  For  example,  one  of  the  key  tasks  omitted  under  the  new 
reorganization  was  that  of  communications  procedures.  According  to  the  instructor  staff, 
when  directed  to  implement  overall  course  reductions  in  order  to  speed  up  the  training 
process,  the  chain  of  command  arbitrarily  decided  that  these  skills  would  be  eliminated 
without  regard  for  the  advice  of  SMEs.  It  was  determined  that  these  skills  could  be 
acquired  on  the  job,  once  the  Marines  reached  their  operational  units.  This  was  clearly 
another  example  of  where  the  SAT  process  was  completely  ignored  and  key  skills  were 
eliminated  in  the  interest  of  saving  time.  Although  the  intent  may  have  been  sound,  this 
action  was  flawed  because  the  advice  and  experience  of  the  SMEs  was  ignored,  thus 
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resulting  in  inadequately  trained  Marines  sent  to  FMF  units.  The  problem  was  confirmed 
by  feedback  from  operational  units,  which  stated  that  many  of  the  field  wiremen,  upon 
reaching  the  fleet,  were  found  lacking  in  basic  skills  normally  expected  of  their  billet  and 
rank  (Lewis,  2000).  As  such,  time  had  to  be  taken  away  from  other  events  by  the 
operational  units  to  train  these  Marines  up  to  the  expected  level  of  proficiency. 

2.  Increased  Training  Requirements 

Another  area  of  concern  within  this  course  dealt  with  the  introduction  of  new 
communications  equipment  within  the  Marine  Corps  inventory.  Many  of  the  new 
systems  being  fielded  will  function  as  key  components  of  vast  communications  networks, 
and  will  be  the  responsibility  of  field  wiremen  to  install,  maintain  and  operate. 
According  to  one  instructor,  “the  days  of  simply  running  wire  and  climbing  telephone 
poles  are  over!”  The  result  of  this  vast  technology  explosion  is  that  new  equipment  is 
being  introduced  as  complements  rather  than  replacements  for  existing  equipment.  A 
review  of  the  Course  Descriptive  Data  found  that  the  students  were  currently  responsible 
for  gaining  proficiency  with  not  less  than  ten  major  sets  of  communications  devices,  to 
include  accompanying  cables,  power  units  and  assorted  support  equipment.  The  number 
of  systems  for  which  field  wiremen  are  now  responsible  is  therefore  increasing  rather 
than  decreasing,  yet  the  response  of  HQMC  to  this  phenomenon  is  to  cut  formal  training 
rather  than  increase  it.  Even  with  the  introduction  of  CBT  into  the  classroom,  the 
instructor  staff  was  in  consensus  that  there  is  still  insufficient  time  currently  available  to 
teach  the  Marines  everything  they  should  know  to  function  proficiently  in  their  basic 
MOS.  Additionally,  it  was  explained  that  due  to  the  reorganization  and  consolidation  of 
certain  MOSs  within  the  communications  field  (2514  with  2512,  for  example),  the 
students  were  now  required  to  learn  additional  skills  that  they  had  previously  not  been 
held  responsible  for  at  that  point  in  their  careers  (Rice,  2000).  Such  requirements  have 
only  compounded  the  problem  of  insufficient  time  to  teach  a  myriad  of  skills  and  tasks. 

3.  Automated  Electronic  Classroom 

There  were  distinctly  mixed  opinions  regarding  the  application  of  CBT  in  the 
classroom.  To  support  the  training  effort,  the  facilities  available  included  an  80-man  and 
50-man  capacity  AEC,  each  providing  individual  workstations  with  self-paced, 


46 


interactive  media  work  packages.  The  older,  more  senior  instructors  felt  the  CBT 
technology  had  some  merit,  but  that  too  much  emphasis  was  being  placed  on  this 
equipment  to  take  the  place  of  traditional  student/instructor  interaction.  In  the  opinion  of 
these  instructors,  overall  lecture  and  demonstration  time  had  been  significantly  reduced, 
impacting  on  student  comprehension.  The  younger  instructors,  however,  found  that 
under  the  present  time  constraints,  the  introduction  of  CBT  into  the  classroom  was  the 
only  feasible  solution  to  successfully  accomplish  their  jobs.  A  review  of  student 
performance,  in  terms  of  test  scores  and  completion  rates,  provided  some  measure  of  how 
the  students  were  reacting  to  the  new  technology.  Test  averages  from  three  random 
classes  that  graduated  prior  to  the  activation  of  the  AEC  (which  occurred  in  early  1999) 
totaled  approximately  89.39  percent,  while  those  of  three  random  classes  that  followed 
totaled  95.14  percent.  Granted  that  this  AEC  application  is  still  in  its  infancy  and 
insufficient  data  currently  exist  for  a  more  thorough  and  detailed  study,  these  early 
indications  support  claims  made  by  various  studies  that  the  use  of  CBT  results  in  no 
significant  change  in  student  comprehension  and  learning  (Russell,  1999  and  Mattoon, 
1997).  It  was  recognized,  however,  that  even  the  junior  instructors  preferred  more  lecture 
and  practical  application  time  when  possible,  finding  that  similar  to  the  LAB  VOLT 
example  in  Company  A,  overall  remediation  rates  decreased,  while  the  rate  of  student 
comprehension  and  progression  increased  proportionately.  It  seems  the  issue  of 
timesaving  though  the  use  of  CBT  and  other  automated  training  technologies  was  being 
addressed  as  previously  discussed.  Rather  than  providing  more  time  to  devote  to  other 
requirements,  the  saved  time  was  simply  being  eliminated  from  the  POI,  requiring  the 
instructors  to  do  more  with  less. 

4.  Advanced  Level  Training 

Advanced  skill  training  was  provided  in  Company  B  in  the  Communications 
Systems  Chief  Course  (CSCC),  and  the  Operations  Communications  Chief  Course 
(OCCC).  The  CSCC  was  conducted  three  times  per  year  with  an  average  throughput  of 
120  Marines  per  year.  The  OCCC  held  eight  classes  per  year,  consisting  of  four  wire  and 
four  radio  sub-courses  for  an  average  annual  throughput  of  216  Marines.  As  part  of 
overall  course  reductions,  the  CSCC  was  shortened  from  12  weeks  to  nine  weeks,  while 
the  OCCC  was  cut  from  16  weeks  to  12  weeks.  The  CSCC  was  significantly  reorganized 
after  the  introduction  of  the  AEC,  as  part  of  the  curriculum  package.  Each  course 
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consisted  of  two  subclasses  focusing  on  radio  and  wire  networks  respectively.  Previously 
the  POI  had  consisted  of  only  a  single  class,  divided  into  thirds  with  an  equal  amount  of 
time  spent  on  radio,  wire  and  data  information  systems.  The  instructors  who  taught  these 
courses  were  the  most  senior  instructors  within  MCCES,  having  a  wealth  of  both 
operational  and  training  experience.  Their  perceptions  of  the  use  of  CBT  within  the 
classroom  were  generally  positive,  although  they  too  were  in  agreement  with  many  of  the 
other  school  personnel  that  the  time  savings  realized  from  the  use  of  automated  training 
technologies  were  simply  being  eliminated,  rather  than  used  for  additional  skill 
development.  Their  observations  were  that  while  the  overall  quality  of  the  students  was 
excellent,  general  experience  levels  were  significantly  lower  due  to  more  rapid  promotion 
practices.  This  situation  thus  required  more  time  to  instill  knowledge  that  in  the  past  the 
average  student,  at  that  stage  of  their  career,  would  have  taken  for  granted.  General 
computer  skills  were  also  found  to  be  lacking  among  students  in  both  the  CSCC  and  the 
OCCC,  requiring  additional  instructor  time  to  familiarize  students  with  the  basics  of  word 
processing  and  graphics  programs.  The  particular  period  of  instruction  that  I  observed 
received  basic  instruction  in  the  use  of  Microsoft  PowerPoint  presentations.  This  skill 
was  a  necessary  part  of  the  curriculum  requirement,  used  to  illustrate  various  diagrams  of 
communications  networks  and  plans.  Based  on  discussions  with  instructor  personnel,  the 
general  student  experience  level  with  such  computer  programs  prior  to  attending  school 
was  moderate,  but  getting  better. 

Similar  reviews  of  test  scores  and  overall  class  averages  both  before  and  after  the 
introduction  of  the  AEC,  were  conducted  to  gain  a  better  perspective  of  student 
performance.  A  random  sampling  of  three  classes  prior  to  the  use  of  the  AEC  resulted  in 
an  overall  student  average  of  92.73  percent,  while  the  cumulative  average  of  three 
random  classes  after  the  use  of  the  AEC  was  found  to  be  91.10  percent.  The  overall 
failure  rate  of  Marines  for  these  particular  courses  was  so  insignificant,  both  before  and 
after  the  use  of  the  AEC,  that  it  was  not  even  considered.  Once  again,  it  is  acknowledged 
that  the  while  the  use  of  the  new  training  technology  is  so  recent  that  sufficient  data  has 
not  yet  been  compiled  for  a  more  detailed  analysis,  the  overall  differences  appear  to  be 
minimal  at  best.  Numerous  other  studies  in  the  field  of  civilian  industry  and  academia 
bear  out  this  “no  significant  difference”  phenomenon  (Russell,  1999). 
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E.  THE  DL  TRAINING  NICHE 

Based  on  the  increased  demands  of  the  advanced  level  courses,  combined  with 
overall  reductions  in  training  time,  instructors  felt  that  future  students  required 
demonstrated  proficiency  in  certain  basic  skills,  prior  to  arrival  at  the  schoolhouse  in 
order  to  be  successful.  Distance  Learning  technologies  were  seen  as  being  most  useful. 
By  reviewing  and  refreshing  certain  core  concepts  and  familiarizing  themselves  with 
basic  computer  processing  skills,  it  was  felt  that  precious  time  spent  on  these  events  could 
be  eliminated.  This  would  allow  for  more  time  devoted  to  advanced  learning.  As  part  of 
this  initiative,  the  staff  was  currently  working  on  a  number  of  initiatives,  to  include  the 
use  of  MCI  courses,  web-based  training  and  other  basic  skill  preassessment  techniques 
that  would  allow  students  to  validate  as  much  of  the  introductory  information  as  possible, 
while  still  at  their  parent  units.  Thus  upon  their  arrival  at  the  schoolhouse,  the  experience 
level  of  the  students  would  be  uniform,  allowing  more  time  to  be  devoted  to  advanced 
concepts  and  techniques.  The  instructors  were  quick  to  point  out  that  in  order  to  be 
successful,  however,  certain  criteria  had  to  be  observed.  First,  the  additional  timesavings 
provided  by  the  use  of  DL  must  not  be  eliminated  from  the  curriculum.  Rather,  they 
should  be  used  to  provide  the  additional  skills  required  for  the  student’s  rank  and 
experience  level.  Second,  prior  to  attending  the  reduced  formal  portion  of  the  course,  the 
prospective  students  must  be  given  adequate  time,  while  still  at  their  units,  to  complete 
the  DL  portion  of  the  instruction  free  from  the  daily  distractions  of  their  regular  duties. 
Failure  to  observe  either  of  these  prescripts  would  not  realize  any  appreciable  savings 
from  the  use  of  DL  technologies. 

F.  SUBJECT  MATTER  EXPERTS 

Another  problem  area  dealt  with  changes  in  course  curricula  based  on  the  infusion 
of  the  new  technology,  both  into  the  classroom  and  into  operational  units.  Similar  to 
students  in  entry-level  courses,  senior  Marines  are  expected  to  be  proficient  with  the  new 
communications  systems  continuously  being  fielded  to  operational  units.  The  principal 
concern  is  from  where  the  training  on  these  new  systems  for  senior  Marines  is  to  come. 
Should  they  be  trained  at  their  units  or  at  the  schoolhouse?  According  to  the  SAT 
process  and  the  more  recent  TEMI,  changes  in  formal  school  course  curricula  must  be 
submitted,  discussed  and  approved  at  a  formal  annual  CCRB.  Subject  Matter  Experts 
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from  the  pertinent  occupational  fields  meet  and  decide  which  ITSs  will  be  taught  at  the 
schoolhouse  and  which  will  be  disseminated  at  the  local  unit  level.  The  problem  is  that 
many  of  the  designated  SMEs  are  not  SMEs  at  all.  Changes  in  technology  and  new 
equipment  occur  so  rapidly  that  many  of  the  senior  Marines  chosen  as  SMEs  are 
unfamiliar  with  new  technologies  and  the  requirements  for  proficiency  with  them.  As 
such,  they  aggressively  push  to  conduct  the  majority  of  the  training  with  this  new 
equipment  at  the  schoolhouse.  These  same  individuals  are  found  to  be  the  ones  who  then 
complain  when  new  Marines  arrive  at  their  units  inadequately  trained  to  operate  the 
equipment.  Schoolhouse  personnel  felt  that  to  properly  deal  with  this  problem,  SMEs 
should  not  be  selected  solely  on  the  basis  of  seniority.  An  individual’s  work  history 
should  be  considered  as  well  (Foy,  2000).  More  importantly,  the  occurrence  of  such 
situations  requires  a  fundamental  change  in  the  way  the  Marine  Corps  views  training.  The 
SAT  and  the  TEMI  are  good  systems,  provided  the  USMC  embraces  them  and  doesn’t 
pay  them  mere  lipservice. 

G.  LEARNING  RESOUCE  CENTERS 

Both  MCCES  and  MCAGCC  operate  and  maintain  LRCs  for  use  by  students  and 
permanent  personnel  alike.  The  AECs  located  at  the  schoolhouse  were  dual  hatted  as 
LRCs  when  not  being  used  for  formal  classes,  while  the  base  maintained  a  functional 
LRC  comprised  of  20  workstations  with  Internet  and  Intranet  capability.  I  was 
specifically  interested  in  looking  at  usage  rates  since  the  introduction  of  these  facilities  in 
mid- 1999.  Reviewing  the  logbook  for  both  LRCs,  I  made  the  following  observations. 

The  principal  categories  of  usage  for  the  LRCs  consisted  of  education-related  topics  such 
as  web-based  MCI  courses,  personal  e-mail  and  general  Internet  applications.  Table  4.1 
illustrates  that  from  the  period  of  June  29,  1999  when  the  facilities  first  opened  to 
December  15,  1999,  the  average  daily  use  rates  in  terms  of  people  per  day  for  the 
MCCES  AEC/LRC  were  as  follows: 


CATEGORY 

Usage  Rate  (People  per  Day) 

1)  Education 

00.45 

2)  Personal  e-mail 

10.27 

3)  General  Internet 

3.00 

Table  4.1  MCCES  AEC/LRC  Average  Daily  Use  Rates  from  29  Jun  99  - 15  Dec  99 


Inspection  of  the  logbook  for  the  base  LRC  revealed  similar  values.  This  period  covers  a 
time  span  of  approximately  62  days,  not  to  include  weekends  and  holidays.  After 
December  15,  1999  personnel  were  no  longer  granted  personal  e-mail  access  due  to 
increased  security  measures  and  bandwidth  redesignation.  For  the  period  from  December 
16,  1999  to  January  19,  2000,  usage  rates  plummeted  to  less  than  one  individual  per  day 
for  both  LRCs.  The  principal  use  of  these  facilities  made  by  personnel  had  apparently 
been  for  personal  e-mail  access.  This  second  finding  came  as  no  real  surprise  for  the 
MCAGCC  LRC,  based  on  initial  research  at  the  schoolhouse  AEC.  An  equally 
contributing  factor  was  a  series  of  directives  issued  by  the  base  Commanding  General, 
who  as  part  of  his  quality  of  life  initiative,  had  previously  ordered  that  the  LRC  be  made 
available  to  all  dependents  to  contact  their  spouses  while  on  deployment. 

Staff  personnel  at  the  schoolhouse  were  not  of  the  opinion  that  the  AEC  would 
receive  much  use  by  students  in  their  off  duty  hours  for  DL  applications.  It  was  felt  that 
the  last  thing  students  would  be  most  likely  to  do  after  a  full  day  of  formal  instruction 
would  be  to  pursue  further  learning  on  their  own.  An  additional  consideration  was  that 
with  reduced  classroom  time,  more  homework  was  being  assigned  than  in  the  older, 
longer  courses.  Most  entry-level  students  are  so  overwhelmed  at  this  point  in  their 
service;  they  simply  have  no  desire  for  career-enhancing  training  unrelated  to  their  MOS. 
It  was  recognized  that  if  DL  applications  had  any  relevance  to  entry-level  training,  it 
would  be  for  Marines  awaiting  training.  Distance  Learning  applications  could  be  used  as 
part  of  the  entry-level  training  requirements  separate  from  the  Marine’s  primary  MOS. 
Competing  distractions  would  be  at  a  minimum.  They  would  be  free  to  concentrate  on 
DL  courses.  As  promising  as  this  initiative  sounds,  no  actions  have  yet  been  taken  to 
implement  it.  MCI  currently  offers  only  three  electronic  courses  of  instruction,  which  are 
as  follows  (Taylor,  2000): 

1)  Land  Navigation 

2)  Fundamentals  of  Diesel  Engines 

3)  Personal  Finance 

Based  on  discussion  with  the  LRC  facilitator,  the  electronic  tests  for  these  courses  do  not 
match  the  course  numbers,  leading  to  confusion  for  the  student  and  the  test  administrator. 
Until  MCI  solves  these  problems  and  converts  more  of  its  courses  to  an  electronic  format, 
which  it  is  currently  doing,  usage  rates  will  remain  low. 
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Perhaps  the  most  disturbing  situation  I  observed  during  my  visit  was  that  T&E 
Branch  is  already  seeking  feedback  on  the  LRC,  although  insufficient  time  has  elapsed 
for  the  system  to  prove  itself.  Fully  operational  for  close  to  nine  months,  the  system  has 
been  underutilized  for  lack  of  instructional  materials.  Once  this  situation  is  corrected,  the 
potential  use  of  this  DL  application  should  increase  for  the  appropriate  category  of  users. 


H.  CHAPTER  SUMMARY 

The  information  gathered  at  MCCES  and  MCAGCC  thus  revealed  some 
interesting  inconsistencies  between  the  plan  outlined  in  the  USMC’s  strategic  vision  for 
the  application  of  advanced  instructional  technologies,  and  what  is  actually  occurring  at 
the  schoolhouse  as  well  as  in  the  operational  units.  Improper  use  of  the  SAT  and  TEMI, 
combined  with  flawed  expectations  of  CBT/DL  technologies,  have  resulted  in  drastic 
reductions  in  course  lengths,  often  to  the  detriment  of  the  quality  of  instruction. 
Increased  training  requirements  caused  by  continuous  fielding  of  new  equipment  have 
further  complicated  already  packed  training  schedules,  which  as  mentioned  above,  are  in 
the  process  of  being  shortened.  Problems  with  the  experience  level  of  SMEs,  chosen  to 
decide  which  tasks  to  teach  at  the  schoolhouse  and  which  to  leave  with  operational  units, 
have  brought  into  question  the  overall  ability  and  quality  of  the  individual  Marine. 
Finally,  a  lack  of  adequate  instructional  materials  has  resulted  in  a  gross  underuse  of  the 
LRCs,  by  both  students  and  permanent  personnel.  As  serious  as  these  problems  are, 
solutions  do  exist  as  evidenced  by  the  recommendations  of  schoolhouse  personnel.  The 
unanswered  question  is  how  long  will  these  problems  be  allowed  to  manifest  themselves 
before  action  is  taken? 
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V. 


ROI  METHODOLOGY 


A.  COST-BENEFIT  ANALYSIS  PROCESS 

CBA  encompasses  a  range  of  procedures  and  is  not  a  single  technique.  Sassone 
and  Schaffer  (1978)  contend  that  although  CBA  incorporates  certain  general  principles,  it 
is  difficult  to  design  an  all-purpose  CBA  procedure  because  of  differences  in 
public/govemment  projects.  They  provide  a  basic  framework  for  conducting  a  CBA 
consisting  of  initial  planning  stages  followed  by  data  collection,  separate  cost  and  benefit 
analyses,  and  presentation  of  results.  McMichael  (1985)  provides  a  similar  framework. 
While  there  are  some  differences  in  their  formulations,  the  authors  would  probably  agree 
with  Swope  (1976)  that  a  CBA  process  should  include  the  following  steps: 

•  Formulate  Assumptions 

•  Determine  Alternatives 

•  Determine  Costs  and  Benefits 

•  Compare  and  Select  Alternatives 

•  Conduct  Sensitivity  Analysis 

Assumptions  are  usually  made  regarding  what  variables  will  affect  the  process 
and  the  range  of  values  those  variables  will  present.  The  alternatives  will  include  the  new 
system  and  one  or  more  other  possibilities.  Frequently,  one  of  these  is  an  existing 
system.  After  the  costs  and  benefits  of  alternatives  have  been  determined,  they  are 
compared  and  a  selection  is  made.  In  CBA,  the  best  alternative  is  the  one  yielding  the 
greatest  net  benefit  (i.e.,  the  alternative  whose  benefit  value,  expressed  in  monetary  terms, 
less  its  cost  is  the  greatest)  (Simpson,  1995).  Orlansky  (1989)  provides  the  following 
concrete  example: 

[In]  cost-benefit  analysis... both  the  input  and  output  values  can  be 
measured  in  monetary  terms.  This  requires  an  open  market  to  assess  the 
value... of  the  output  that  results  from  a  particular  use  of  resources  (i.e., 
the  costs).  One  example  might  be  a  cost-benefit  analysis  of  a  particular 
form  of  advertising.  The  costs  are  those  needed  to  develop  and  conduct  a 
particular  advertising  program;  the  benefits  are  the  profits  that  may  be 
attributed  to  the  advertising  program  (p.  ix). 
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Assumptions  are  required  in  planning  a  CBA  and  these  can  lead  to  uncertainty  in 
the  outcomes  of  analyses.  If  the  CBA  is  locked  into  a  single  set  of  assumptions  with  the 
intent  of  obtaining  a  definitive  result,  its  outcome  may  be  too  fragile  to  be  trustworthy.  It 
is  more  sensible  to  vary  the  assumptions  systematically  and  to  provide  the  results  of 
analyses  under  different  assumptions.  This  procedure  is  referred  to  as  sensitivity  analysis. 

According  to  Simpson  (1995),  a  CBA  does  not  sufficiently  evaluate  military 
projects,  however,  for  there  is  no  market  available  to  establish  the  monetary  value  of  the 
output  in  terms  of  performance,  although  inputs  can  be  expressed  in  monetary  terms.  He 
further  states  that  a  cost-effectiveness  analysis  (CEA)  is  the  preferred  method  used  in  the 
DoD  to  make  decisions  regarding  alternative  courses  of  action  where  the  outcomes  affect 
military  performance.  Examples  may  include  choosing  among  a  set  of  the  following: 

a.  Weapons  Systems 

b.  Weapon  System  Upgrade  Programs 

c.  Training  Methods 

A  definition  of  CEA  analogous  to  that  given  earlier  for  CBA  might  be  an  estimation  and 
evaluation  of  the  military  value  associated  with  alternatives  for  achieving  defined  military 
goals  (Simpson,  1995).  CEA  is  used  to  help  meet  military  goals  (rather  than  CBA’s 
public/govemment  goals).  CEA,  like  CBA,  compares  alternatives  using  a  formal  process. 
Criteria  decide  the  outcome  for  both  CEA  and  CBA,  but  the  criteria  differ  (i.e.  military 
value  for  CEA  and  public  benefits  for  CBA)  (Simpson,  1995).  Economic  Analysis ,  an 
additional  term  used  in  a  number  of  DoD  publications,  has  a  meaning  synonymous  with 
CEA  (Rankin  &  Swope,  1991). 

B.  COSTS  AND  MILITARY  VALUE 

Simpson  (1995)  maintains  that  the  costs  of  alternatives  in  a  CEA  are  estimated  in 
a  manner  similar  to  that  of  a  CBA  by  using  cost  models  that  take  into  account  all  of  the 
associated  costs  of  the  alternatives  through  a  projected  life  cycle.  He  suggests,  however, 
that  estimating  military  value  for  a  CEA  is  different  from  estimating  public  benefits  in  a 
CBA.  An  important  difference  between  CEA  and  CBA  is  that  the  outcome  (military 
value)  is  not  defined  in  the  same  terms  as  cost  (Orlansky,  1989;  Rankin  &  Swope,  1991). 
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The  military  value  of  ineffective  training  methods,  for  example,  is  hard  to  monetize.  An 
instructional  program  that  sends  a  poorly  trained  Marine  to  an  operational  unit  represents 
an  opportunity  cost  to  that  unit  in  terms  of  lost  output.  A  senior,  more  experienced 
Marine  must  now  be  detailed  from  his  regular  duties  to  work  with  the  junior  Marine  to 
bring  the  individual  up  to  the  desired  level  of  knowledge  and  skill.  The  time  and  effort 
spent  working  with  the  junior  Marine  is  a  cost  to  the  unit  in  terms  of  the  senior  Marine’s 
normal  output  as  well  as  the  junior  Marine’s  inefficient  output.  The  benefit  of  what 
should  be  a  high-quality  training  program  has  now  become  a  cost.  Yet  how  exactly  does 
one  associate  a  value  with  this  cost  (i.e.,  the  poor-training  program)?  According  to 
Boardman,  Greenburg,  Vining,  and  Weimer  (1996),  the  intermediate  good  method  can 
be  used  to  estimate  the  benefit  of  a  project  (the  training  program)  based  on  its  value 
added  to  the  downstream  activity  (i.e.,  a  qualified  Marine’s  output  in  the  unit). 
Investment  in  the  skills  and  abilities  of  human  beings  improves  the  stock  of  human 
capital,  and  thus  make  individuals  more  productive  and  more  valuable  to  the  organization 
(Boardman,  et  al.,  1996).  In  terms  of  a  cost  analysis,  the  military  value  of  the 
intermediate  good  (the  poor  training  program)  has  become  an  opportunity  cost  in  regard 
to  both  the  junior  Marine’s  lost  output,  and  the  senior  Marine’s  normal  lost  output,  which 
are  relevant  factors  that  are  difficult  to  measure.  The  authors  argue  that  when  dealing 
with  such  intermediate  goods  whose  linkage  to  preferences  is  not  clear,  a  CEA  is  a 
suitable  alternative  to  a  CBA.  Orlansky  (1989)  said: 

[The  cost-benefit]  procedure  cannot  be  followed  when  examining 
the  products  of  a  military  weapon  or  training  program  (emphasis  added). 

There  is  no  open  market  that  can  establish  the  monetary  value  of  increased 
readiness,  better-trained  personnel,  or  better  weapons  (p.  ix). 

Simpson  (1995)  concludes  that,  ultimately  military  value  is  reflected  in  the  degree 
of  combat  success.  According  to  his  analogy;  weapon  system  A  has  greater  military 
value  than  weapon  system  B  if  A  is  more  likely  to  prevail  in  battle  than  B.  Or,  if  two 
training  alternatives  are  being  compared,  treatment  A  has  greater  military  value  than 
treatment  B  if  A  better  equips  students  to  prevail  in  battle  than  B.  Military  value  can  be 
assessed  empirically  only  in  combat  and  it  is  impractical  to  wait  for  a  war  to  make  an 
assessment.  An  alternative  to  combat  is  to  create  a  combat-like  environment  (e.g.,  to  use 
an  instrumented  live  exercise).  In  performing  CEA,  measures  of  effectiveness  (MOE)  are 
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used  that  ostensibly  predict  combat  success  (Simpson,  1995).  In  the  experimental 
paradigm,  Simpson  (1995)  further  defines  MOEs  as  equivalent  to  dependent  variables; 
the  same  variables  used  to  assess  the  impact  of  an  experimental  treatment  condition. 

Like  CBA,  CEA  encompasses  a  wide  range  of  procedures  and  is  not  a  single 
technique.  Because  of  conceptual  similarities  between  CBA  and  CEA,  it  is  reasonable  to 
extend  Sassone  and  Schaffer’s  (1978)  contention  regarding  the  difficulty  of  designing  an 
all-purpose  CBA  procedure  to  the  realm  of  CEA.  Likewise,  the  basic  framework  for 
conducting  a  CEA  parallels  that  of  a  CBA,  described  by  Swope  (1976),  but  with  a  slight 
change  to  the  third  step  (“Benefits”  becomes  “Military  Value”): 

•  Formulate  Alternatives 

•  Determine  Alternatives 

•  Determine  Costs  and  Military  Value 

•  Compare  and  Select  Alternatives 

•  Conduct  Sensitivity  Analysis 

Since  cost  and  military  value  use  different  units,  selection  of  alternatives  cannot  be  done 
on  a  cost  basis  alone  as  with  CBA  (Simpson,  1995).  Orlansky  (1989)  has  described  the 
decision-making  logic  as  illustrated  below: 


COST 

EFFECTIVENESS 

LESS 

SAME 

MORE 

LESS 

UNCERTAIN 

ADOPT 

ADOPT 

SAME 

REJECT 

UNCERTAIN 

ADOPT 

MORE 

REJECT 

REJECT 

UNCERTAIN 

Table  5.1  Orlansky’s  Decision  Logic  Diagram  for  Evaluating  the  Relative  Effectiveness  and 


Cost  of  Two  Training  Methods  During  CEA  (1989) 

Orlansky  (1989)  commented  as  follows  on  the  interpretation  of  the  diagram. 

a.  If  one  alternative  is  as  effective  or  more  effective  than  another  mid  it 
costs  less,  adopt  it;  it  is  also  the  preferred  choice  if  it  is  more  effective 
and  costs  the  same. 

b.  If  an  alternative  is  less  effective  and  costs  the  same  or  more  than 
another  to  which  it  has  been  compared,  reject  it;  this  is  also  the  case  if 
it  is  equally  effective  but  costs  more. 

c.  If  any  of  the  following  combinations  of  the  cost  and  effectiveness  of  an 
alternative  is  found,  no  rational  preference  can  be  made; 
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(1)  less  effective  and  less  cost 

(2)  equal  effectiveness  and  equal  cost 

(3)  more  effective  and  more  cost 


Based  on  the  nature  of  the  data  gathered  from  my  observations  at  MCCES,  the 
characteristics  of  CEA  are  particularly  relevant  here,  so  a  combination  of  both  CBA  and 
CEA  will  be  used  to  conduct  my  analysis. 

C.  BENEFITS  AND  VALUE  ADDED 

Measuring  the  direct  benefits  or  results  of  instruction  is  a  more  complex  task  than 
cost  identification  since  some  benefits  may  be  difficult  to  identify.  The  objective  is  to 
attach  a  quantitative  value  to  each  identified  benefit,  which  must  be  derived  from  the 
instructional  alternative  being  examined.  There  may  be  other  factors  that  contribute  to  a 
given  benefit.  The  analyst  must  isolate  and  measure  only  the  effects  of  training.  The 
American  Society  for  Training  and  Development’s  2000  ASTD  State  of  the  Industry 
report,  which  outlines  trends  in  over  60  major  industries,  lists  the  following  as  typical 
benefits  to  consider  when  conducting  an  analysis  of  DL  technology: 

•  Reduced  training  time.  Reductions  may  be  realized  in  facility, 
personnel  and  travel  costs,  as  well  as  less  nonproductive/lost  time 
while  the  student  is  in  training. 

•  Reduced  material  costs.  Potential  cost  savings  for  revision, 
distribution  and  maintenance  of  instructional  materials/courseware  in 
electronic  format  versus  printed  or  “hard  copy”  materials. 

•  Improved  on-the-job  safety  and  proficiency:  Translates  into  fewer 
accidents  and  increased  productivity. 

•  Improved  job  performance.  Less  time  to  perform/accomplish  tasks, 
increased  productivity/efficiency/quality  of  work,  or  improved 
management/command/supervision/decision-making  ability,  amount 
of  positive  versus  negative  feedback  from  “clients”,  changes  in 
employee  morale  and  motivation. 

•  Reduced  manpower  requirements.  Potential  manpower  savings  as  a 
result  of  increased  efficiency  or  productivity,  less  supervision  required, 
etc. 
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•  Reduced  equipment  requirement.  Simulation  training  may  reduce 
the  need,  impact,  or  expense  of  allocating  actual  equipment  to  train 
hands-on  skills. 

•  Higher  equipment  operational  availability.  Fewer 

operator/maintainer  errors,  safety  incidents/accidents,  etc.,  may  result 
in  lower  maintenance  requirements  and  equipment/system  downtime. 

Reduced  to  its  most  elemental  form,  a  CBA  evaluates  costs  versus  benefits  for  a 
particular  project.  Boardman,  et  al.,  (1996),  state  that  the  basic  decision  rule  for  a  single 
alternative  is  simple:  Add  up  the  present  value  of  benefits  (B),  add  up  the  present  value 
of  costs  (C),  and  see  which  is  larger.  If  benefits  exceed  costs,  then  proceed  with  the 
project.  If  not,  stay  with  the  status  quo.  In  short,  the  analyst  should  recommend 
proceeding  with  the  project  based  on  the  following  formula: 

NPV  =  B-C>  0 

When  there  is  only  one  potential  project,  proceed  with  that  project  if  the  net  present  value 
of  benefits  (NPV)  is  positive.  When  there  is  more  than  one  alternative  to  the  status  quo, 
the  rule  is  slightly  more  complicated:  Select  the  project  with  the  highest  NPV  (Boardman, 
et  al.,  1996). 

As  straightforward  as  this  process  may  seem,  however,  problems  arise  in 
determining  precisely  what  the  benefits  will  be,  and  more  particularly  in  attaching  a 
monetary  figure  to  those  benefits  that  are  identified.  Thompson  and  Strickland  (1995), 
address  this  issue  by  incorporating  the  concept  of  a  value  chain  into  the  process  of  cost 
analysis.  According  to  them,  the  primary  analytical  tool  of  strategic  cost  analysis  is  a 
value  chain  identifying  the  separate  activities,  functions  and  business  processes 
performed  in  designing,  producing,  marketing,  delivering  and  supporting  a  product  or 
service.  The  chain  starts  with  raw  materials  supply  and  continues  on  through  parts  and 
components  production,  manufacturing  and  assembly,  wholesale  distribution  and 
retailing  to  the  ultimate  end  user  of  the  product  or  service  (Thompson,  et  al.,  1995).  In 
our  case,  the  raw  material  is  the  entry-level  Marine  attending  basic  skills  training  and  the 
senior  Marine  attending  advanced-level  training.  The  ultimate  end  user  is  the  Marine 
Corps,  or  more  specifically,  the  operational  units.  The  key  here  is  to  incorporate  DL 
technologies  in  such  a  way  as  to  increase  the  value  of  the  Marine  by  enhancing  his 
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learning  ability  and  retention  skills,  while  simultaneously  reducing  the  amount  of  time 
the  Marine  is  away  from  the  operating  forces. 

D.  ANALYSIS  STRATEGY 

Continuing  on  with  Simpson’s  (1995)  steps  of  conducting  a  CEA,  I  will  now 
apply  them  to  my  observations  from  MCCES. 

1.  Formulate  Assumptions 

My  assumptions  are  as  follows: 

a.  All  expenditures  on  the  hardware  and  software  components  of 
MarineNet  for  the  active  duty  force  are  considered  sunk  costs,  as  the 
money  has  already  been  allocated  and  spent.  The  system  is  in  place, 
and  may  be  considered  fully  operational. 

b.  I  found  the  RDC  at  MCCES  to  be  fully  manned  and  operational,  and 
am  therefore  including  course  conversion  costs  as  sunk  costs. 

c.  Due  to  time  constraints  on  the  scope  of  this  research,  the  impact  of 
MarineNet  and  DL  technologies  on  the  Reserve  component  will  not  be 
considered. 

d.  Based  on  my  observations  at  MCCES,  I  will  only  consider  DL 
applications  to  advanced-skill  level  training. 

e.  I  will  apply  the  industry  standard  of  an  estimated  30  percent  reduction 
in  training  time  due  to  the  application  of  DL  technologies. 

f.  Based  on  discussions  with  schoolhouse  personnel,  current  course 
reductions  did  not  include  calculated  savings  based  on  the  use  of  DL 
technologies. 

2.  Determine  Alternatives 

The  alternative  to  incorporating  the  proposed  DL  initiatives  would  obviously  be 
the  status  quo,  since  it  would  make  no  sense  to  dismantle  the  system  that  is  currently  in 
place.  The  Marine  Corps  has  committed  itself  to  the  use  of  MarineNet,  and  has  no  choice 
but  to  continue  down  that  path.  To  start  from  scratch  on  a  completely  new  program 
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would  result  in  a  net  loss  on  all  previous  network  investment.  The  remaining  question  is 
how  DL  technologies  may  be  applied  to  the  advances  already  in  place  to  realize 
additional  benefits  in  training  and  education.  Although  considerable  reductions  in  course 
lengths  have  already  taken  place,  these  have  been  instituted  without  adequate 
consideration  of  the  use  of  DL  technologies.  The  focus  up  to  this  point  has  been  more  on 
increased  classroom  efficiency  through  automation,  rather  than  improvements  in 
preassessment  and  validation  techniques. 

3.  Determine  Costs  and  Military  Value 

My  initial  computations  address  only  cost  reductions  that  have  been  attributed  to 
CBT,  but  later  consider  the  additional  gains  that  may  be  realized  by  the  inclusion  of  DL 
applications.  I  also  attempt  to  focus  on  a  more  intangible  aspect  of  these  changes, 
involving  the  impact  upon  the  military  value  of  the  instruction.  Table  5.2  represents  the 
current  reductions  that  have  already  taken  place  in  the  reviewed  courses.  The  figures  in 
the  Training  Input  Plan  (TIP)  column  represent  original  course  lengths  prior  to  the 
implemented  reductions  that  are  reflected  in  the  schoolhouse  CDDs.  The  data  were 
compiled  from  a  series  of  interviews  with  personnel  at  T&E  Division,  HQMC,  as  well  as 
instructors  from  MCCES. 


Course 

Identification 

COURSE  LENGTH  (Days) 

30% 

Actual 

Name 

Number 

TIP 

(Old) 

-jagTWMlIfe 

Reduction  (Days) 

Reduction  (%) 

BEC 

M092721 

93 

40 

65 

57 

TTC 

M09TA31 

112 

80 

78 

29 

BFWC 

M092471 

42 

17 

29 

60 

OCCC 

M0925A1 

118 

60 

83 

49 

CSCC 

M09CHK1 

86 

45 

60 

48 

Table  5.2  Current  Reductions  in  Selected  Courses  at  MCCES 


Based  on  the  figures  in  the  above  table,  it  can  clearly  be  seen  that  the  current  reductions, 
which  are  attributable  to  the  inclusion  of  CBT  and  improvements  in  instructional 
efficiency,  have  already  exceeded  the  estimated  industry  standard  of  30  percent  that  is 
associated  with  the  application  of  DL  technologies.  Unfortunately  no  DL  applications 
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have  yet  been  instituted,  leaving  one  to  wonder  if  further  reductions  are  feasible.  Based 
on  discussions  with  instructors  at  MCCES,  many  of  these  current  reductions,  directed  by 
HQMC,  were  excessive,  arbitrary,  and  sacrificed  skills  critical  to  the  quality  of  the 
training.  They  feel  that  further  reductions  could  prove  imprudent.  Examined  at  face 
value,  one  can  only  wonder  how  courses  of  instruction  as  technical  and  complex  as  the 
ones  reviewed  above  can  be  cut  so  drastically  and  still  produce  a  qualified  Marine.  This 
leads  back  to  the  concept  of  estimating  military  value.  Are  these  current  course 
reductions  based  solely  on  cost  reduction/avoidance,  or  is  due  consideration  being  given 
to  the  finished  product  (i.e.,  the  skill  level  of  the  Marine?)  According  to  personnel  at 
MCCES,  such  is  not  the  case.  Arbitrary  reductions  combined  with  failure  to  follow  the 
provisions  of  the  SAT  and  TEMI  have  been  the  norm  rather  than  the  exception, 
significantly  impacting  on  the  military  value  of  the  courses. 

Table  5.3  represents  data  collected  from  both  T&E  Division,  DL  Branch,  as  well 
as  from  instructors  at  MCCES.  The  model  is  an  exact  replica  of  one  used  by  T&E 
Division  to  calculate  potential  savings  based  on  the  application  of  DL  technologies.  I 
was  not  able  to  transcribe  the  data  directly,  however,  as  the  figures  used  for  the  course 
lengths  were  in  some  cases  outdated.  The  course  lengths  used  in  my  table  come  directly 
from  the  CDDs  at  MCCES.  The  number  of  Man-Days  is  calculated  by  multiplying  the 
yearly  course  input  by  the  course  length.  Based  on  the  fact  that  there  is  no  single,  proven 
formula  for  determining  the  cost  of  training  a  Marine  per  day,  and  also  in  consideration 
that  many  military  units  use  outdated/unknown  cost  factors,  I  analyzed  the  data  from 
T&E  Division  to  see  what  I  could  find.  Interestingly  enough,  I  found  that  by  dividing  the 
total  cost  of  each  course  after  incorporating  the  30  percent  reduction,  by  the  number  of 
new  Man-Days  (which  also  included  the  30  percent  reduction),  I  came  up  with  a  constant 
dollar  figure  of  $82.00  per  day.  This  value  remained  approximate  for  all  63  MOSs  within 
the  data  set. 
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Course 

Yearly  Input 

Current  Length 
(Days) 

Annual 

Man-Days 

Man-Day  Cost 

Annual  Course 
Cost  (Man-Days) 

BEC 

1356 

40 

54,240 

$82 

$4,447,680 

TCC 

188 

80 

15,040 

$82 

$1,233,280 

BFWC 

596 

17 

10,132 

$82 

124 

60 

7440 

$82 

$610,080 

230 

45 

10,350 

$82 

$848,700 

Table  5.3  Annual  Cost  Computations  of  Selected  Courses  at  MCCES 


Based  on  these  figures,  Table  5.4  illustrates  the  additional  reductions  that  could  be 
realized  if  the  potential  savings  associated  with  DL  applications  were  applied.  Potential 
savings  from  DL  applications  will  not  be  applied  to  personnel  in  entry-level  courses,  but 
instead  will  be  considered  for  individuals  in  MAT  status. 


Course 

Current  Man- 
Days 

Annual  Course 
Cost 

(Man-Days) 

New  Man-Days 
@30% 
Reduction 

New  Course 
Cost  @  30% 
Reduction 

Annual 

Savings 

BEC 

54,240 

$4,447,680 

TTC 

15,040 

$1,233,280 

10,528 

$863,296 

$369,984 

BFWC 

10,132 

$830,824 

OCCC 

7440 

$610,080 

5208 

$427,056 

$183,024 

CSCC 

10,350 

$848,700 

7245 

$594,090 

$254,610 

Table  5.4  Cost  Analysis  Savings  of  Selected  Courses  at  MCCES  based  on  use  of  DL  Technology 


The  calculations  are  based  on  a  simple  reduction  of  the  current  number  of  Man-Days  for 
the  appropriate  course  based  on  the  industry  standard  of  30  percent.  This  revised  figure  is 
then  multiplied  by  the  $82/day  cost  per  student-day  value  to  attain  the  new  course  cost, 
which  is  then  subtracted  from  the  old  cost  to  obtain  the  annual  savings. 


4.  Compare  and  Select  Alternatives 


As  seen  in  the  preceding  figures,  a  comparison  of  the  status  quo  and  revised 
methods  of  instruction  illustrates  that  savings  in  overall  course  costs  can  be  realized 
through  the  implementation  of  DL  technologies.  The  infrastructure  already  exists  and  the 
personnel  are  in  place  to  convert  portions  of  these  courses  to  a  DL  format.  The  key  issue 
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here  is  to  ensure  that  the  proper  tasks  that  easily  lend  themselves  to  conversion  are  the 
ones  that  are  converted,  while  the  core  competencies  critical  to  the  schoolhouse 
environment  remain  part  of  the  conventional  course.  To  properly  accomplish  this  will 
entail  the  participation  and  agreement  of  SMEs  both  from  the  schoolhouse  and 
operational  units.  To  do  otherwise  will  worsen  the  current  situation  in  which  course 
reductions  have  been  arbitrary  and  drastic,  with  the  result  of  inadequately  trained 
Marines. 


5.  Conduct  Sensitivity  Analysis 

A  properly  conducted  sensitivity  analysis  would  entail  picking  the  right  variables 
to  vary  for  effect  on  outcomes.  It  would  involve  a  review  of  all  tasks  and  skills 
associated  with  each  individual  course  by  SMEs.  More  tasks,  while  lending  themselves 
to  a  DL  environment  would  allow  for  greater  reductions  in  course  length,  conversely 
more  tasks  requiring  personal  interaction  would  lengthen  or  at  least  maintain  course 
curricula  at  current  levels.  Once  again,  the  critical  issue  here  is  adequate  participation  by 
SMEs  from  both  the  schoolhouse  and  operational  units,  who  follow  the  precepts  of  the 
SAT  and  TEMI.  A  further  consideration  is  that  the  time  savings  realized  from  DL 
technologies  are  not  immediately  translated  into  proportional  reductions  in  classroom 
time,  without  first  ensuring  that  all  critical  tasks  and  skills  are  being  taught  in  one  form  or 
another.  The  significant  reductions  in  some  of  the  courses  discussed  earlier  are  evidence 
that  insufficient  consideration  was  given  to  the  final  ability  level  of  course  graduates. 

Another  variable  that  could  be  changed  as  a  part  of  this  analysis  would  be  student 
throughput,  based  on  a  review  of  the  TIP.  The  TIP  is  produced  in  relation  to  a  stated 
Fiscal  Year  (FY),  and  covers  one  year  from  execution  and  four  “out-years”  for  planning. 
The  number  of  students  predicted  for  each  particular  course  are  based  on  validated 
training  requirements  for  each  fiscal  year,  necessary  to  obtain  or  maintain  desired 
manning  levels  (TIP  User’s  Guide,  1999).  This  value  is  thus  an  important  input  to  the 
TIP  process  that  is  the  basis  for  manpower  and  budgeting  costs.  If  this  value  is 
overestimated,  funds  may  be  over-obligated  and  not  used.  If  underestimated,  special 
efforts  are  necessary  to  fund  and  obtain  unplanned  school  seats.  Personnel  responsible 
for  these  estimates  (i.e.,  SMEs  and  Occupational  Field  Managers)  are  therefore  extremely 
conscientious  and  meticulous  in  their  calculations.  These  values  thus  remain  constant 
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and  are  not  subject  to  significant  change,  as  seen  in  the  close  approximation  between  the 
courses  I  observed  and  the  corresponding  estimations  in  the  TIP. 

A  final  variable  that  could  be  changed  is  the  estimated  cost  per  student-day.  This 
value  has  been  found  to  be  quite  nebulous,  and  in  many  cases  totally  unrelated  to  the 
situation  for  which  it  is  being  used. 

E.  ROI  MODEL 

Table  5.5  illustrates  the  ROI  process  model  used  by  the  ASTD  (1998)  where 
sequential  steps  simplify  a  potentially  complicated  method.  This  ROI  model  provides  a 
systematic  approach  to  ROI  calculations. 


This  step-by-step  approach  keeps  the  process  manageable  so  that  users  can  tackle  one 
issue  at  a  time.  The  model  also  emphasizes  that  this  is  a  logical  process  that  flows  from 
one  step  to  another.  Applying  the  model  from  one  ROI  calculation  to  another  provides 
consistency,  understanding,  and  credibility  (ASTD,  1998). 

Data  collection  is  obviously  the  starting  point  and  most  central  aspect  of  the  ROI 
process.  As  illustrated  in  Table  5.5,  items  in  the  circles  are  issues  that  must  be  addressed 


when  determining  the  specific  data  collection  method.  This  consideration  proved 
particularly  challenging  for  my  research  as  no  data  supporting  the  use  of  DL  technology 
currently  exist.  As  such,  those  tangible  factors  discussed  in  Chapter  III  were  considered 
first.  As  indicated  in  my  assumptions,  I  considered  course  conversion  costs,  support 
costs  and  resource  requirements  as  sunk  costs  since  the  MarineNet  infrastructure  is 
already  in  place  and  operational  at  MCESS  and  MCAGCC.  The  funds  allocated  for  this 
project  have  already  been  spent  and  are  therefore  not  a  consideration  in  the  effects  of  DL 
technology  applications.  Easily  quantifiable  factors  that  do  impact  on  a  cost  analysis 
include  timesavings,  reduced  travel  costs,  and  increased  course  efficiency.  Based  on  a 
study  conducted  by  Bowes  (1991)  on  the  average  costs  of  training  first-term  Marines, 
Table  5.6  lists  values  for  these  factors  and  combines  them  as  Operation  and  Maintenance 
(O&M)  costs,  student  pay,  and  costs  of  military  instructors. 


Course 

Direct 

Costs 

Indirect 

Costs 

Student 

Pay 

Cost  of  Student 
per  Course 

Cost  of  Instructor 
Time/Course 

Total  Cost  of 
Course 

TTC 

$1256 

$547 

$11516 

$13319 

$2616 

CSCC 

$631 

$368 

$6380 

$7379 

$6670 

$14,049 

OSCC 

$1313 

$768 

$13,359 

$15,440 

$6045 

$21,485 

Table  5.6  Training  Costs  for  Individual  Marines  from  Selected  Courses  at  MCCES,  from  Bowes 
(1991) 


Based  on  these  total  cost  values  (in  1991  dollars)  and  the  predicted  benefits  listed  in 
Table  5.4,  a  benefit  cost  ratio  (BCR)  and  a  figure  for  ROI  can  now  be  calculated  for  each 
of  the  selected  courses.  Table  5.7  presents  these  values  based  on  the  following  formulas: 


BCR  =  Total  Benefits 
Program  Costs 


ROI  =  Total  Benefits  —  Program  Costs 
Program  Costs 


Course 

Total  Projected  Annual 
Benefits  per  Course 

Total  Annual 
Course  Costs 

BCR 

ROI  (%) 

TTC 

$369,984 

$159,350 

2.32 

132% 

CSCC 

$254,610 

$112,392 

2.27 

127% 

OSCC 

$183,024 

$64,455 

2.84 

184% 

Table  5.7  BCR  and  ROI  Calculations  on  Selected  Courses  from  MCCES  Based  on  DL  Technologies 


Based  on  the  above  computations,  for  every  dollar  invested  in  DL  applications  for  the 
TTC,  $2.32  would  be  returned.  Likewise  values  of  $2.27  and  $2.84  could  be  recognized 
for  every  dollar  of  DL  technology  invested  in  the  CSCC  and  OSCC  respectively.  In 
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terms  of  the  ROI  calculation,  for  every  dollar  of  DL  technology  invested,  the  dollar 
would  be  recovered  and  another  $1.32,  $1.27,  and  $1.84  would  be  produced  for  the  TTC, 
CSCC,  and  OSCC,  respectively.  When  extrapolated  over  several  years  and  applied  to 
other  appropriate  courses,  the  benefits/cost  savings  of  DL  technologies  become  even 
more  significant.  Unfortunately  further  research  into  those  specifics  is  beyond  the  scope 
of  this  paper,  but  definitely  merit  further  research. 

1.  Value-added  Activities 

Another  significant,  yet  somewhat  intangible  factor  that  merits  further 
consideration  is  the  impact  of  DL  technology  on  unit  readiness.  Unit  readiness  is 
currently  measured  by  the  Status  of  Resources  and  Training  Systems  (SORTS)  report 
(Marine  Corps  Manpower  System  Brief,  1998).  The  SORTS  report  has  three  principal 
indexes  comprised  of  personnel  readiness,  equipment  readiness,  and  training  readiness. 
The  first  two  categories  are  relatively  easy  to  measure  in  terms  of  readiness.  Either  a  unit 
has  the  authorized  numbers  of  personnel  and  equipment  or  it  doesn’t.  Either  the  assigned 
equipment  works  or  it  doesn’t.  Accessing  the  training  readiness  of  the  assigned  personnel 
is  another  matter  entirely.  Although  it  can  readily  be  determined  whether  the  assigned 
personnel  have  been  to  the  requisite  schools  for  their  rank  and  billet,  it  is  much  more 
difficult  to  determine  the  amount  of  knowledge  and  skills  retained  from  such  training. 

DL  has  enormous  potential  in  this  area.  In  terms  of  unit  readiness,  you  would  have  more 
senior  personnel  in  the  operational  units  for  more  of  their  obligated  time,  who  now  have 
the  same,  if  not  better,  opportunities  to  receive  instruction  on  the  most  current 
information  put  out  by  the  schoolhouse.  Moreover,  when  the  time  does  arrive  for  those 
senior  personnel  to  attend  a  formal  school,  they  can  refresh  many  of  their  basic  skills 
prior  to  attending  school  while  still  at  their  parent  units.  The  result  of  DL  applications 
will  be  a  much  more  informed  and  up  to  date  student  who  can  proceed  through  the  formal 
instruction  at  a  much  faster  rate,  with  greater  comprehension  and  retention.  Finally,  for 
those  personnel  unfortunate  enough  not  to  be  chosen  to  attend  formal  school,  DL 
technologies  will  allow  them  to  stay  current  on  the  most  recent  changes  in  their  respective 
MOS.  From  this  perspective,  DL  technologies  can  be  further  viewed  as  a  value-added 
activity  that  although  hard  to  quantify,  improves  the  overall  quality  and  ability  of  the 
learner. 
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2.  Indirect  Benefits 


Several  of  the  less  tangible  benefits  discussed  by  Mattoon  (1998)  equally  apply. 
Based  on  observations  and  discussions  at  MCCES,  DL  technologies  are  favorably 
disposed  toward  the  factors  of  acceleration,  automation,  availability,  generalizability, 
longevity  (through  reinforcement),  stability,  strategics,  and  transfer.  Although  difficult  to 
quantify,  indirect  benefits  such  as  these  warrant  consideration  when  assessing  the  total 
value  that  DL  technologies  can  make  to  training  and  education  programs.  Furthermore, 
the  three-level  MOE  model  used  by  CNET  is  equally  applicable  here  and  would  most 
definitely  provide  a  favorable  return  on  DL  applications.  As  supported  by  observations  at 
MCCES,  the  timesavings  MOE  has  already  occurred  through  the  use  of  the  AEC  in 
which  substantial  reductions  in  attrition  and  setbacks  were  evidenced  in  the  LAB  VOLT 
example.  Follow-on  planning  is  already  underway  to  apply  the  lessons  of  this  successful 
experiment  to  a  DL  environment  that  may  be  exploited  by  the  reserve  component 
(Herring,  2000). 

Phillips  (1997)  states  that  intangible  measures  are  the  benefits  or  detriments 
directly  linked  to  the  training  program,  which  cannot  or  should  not  be  converted  to 
monetary  values.  He  maintains  that  these  measures  are  often  monitored  after  the  training 
program  has  been  conducted  and,  although  not  converted  to  monetary  values,  are  still 
very  important  in  the  evaluation  process.  A  variety  of  available  indirect  benefits  reflect 
the  success  of  any  training  and  education  program.  Although  they  may  not  be  perceived 
as  valuable  as  specific  monetary  measures,  they  nevertheless  are  an  important  part  of  an 
overall  evaluation.  Phillips  (1997)  further  suggests  that  intangible  measures  should  be 
identified,  explored,  examined,  and  monitored  for  changes  when  they  are  linked  to  the 
program.  Collectively,  he  concludes,  they  add  a  unique  dimension  to  the  overall  program 
results  since  most,  if  not  all,  programs  have  intangible  measures  associated  with  them. 

F.  CHAPTER  SUMMARY 

According  to  Derryberry  (1998),  many  organizations  are  beginning  to  require 
inclusion  of  a  CBA  or  ROI  analysis  to  justify  an  expenditure  such  as  what  is  required  for 
a  performance  improvement  environment  or  electronic  learning  system.  Simply  stated, 
ROI  methods  are  used  to  demonstrate  that  the  value  of  the  benefits  realized  from  using 
such  a  solution  is  going  to  exceed  the  price  of  its  development  and  implementation.  She 


67 


further  implies  that  ROI  and  technology  implementations  clearly  go  hand  in  hand. 
Organizations  are  thus  looking  for  new  ways  to  leverage  existing  and  anticipated 
technology  investments.  Consequently,  it  is  important  to  remember  that  in  applied 
settings,  ROI  is  particularly  valuable  as  a  means  to  the  end  of  making  effective  business 
decisions.  In  applied  settings,  research-styled  methodological  rigor  is  less  the  point  than 
developing  a  well-supported  rationale  for  ensuring  that  resources  are  being  allocated  in 
economical  ways  (Derryberry,  1998). 

Even  so,  Derryberry  (1998)  continues,  ROI  is  not  simply  evaluation  methodology 
focused  solely  on  financial  concerns.  Technology  decisions  are  often  not  just  about  the 
financial  bottom  line.  Even  if  data  suggest  that  a  low-end  non-technological  solution 
may  yield  positive  results,  there  may  be  a  variety  of  organizational  reasons  that  this  may 
not  be  as  attractive  to  an  organizational  funder  as  is  a  more  sophisticated,  costly 
technology  solution  (Derryberry,  1998).  She  concludes  that  while  ROI  methods  provide 
structure  for  systematically  collecting,  organizing  and  compiling  data,  the  interpretation 
of  those  data,  and  the  resulting  perception  of  value  those  data  represent,  may  be  highly 
subjective. 

Phillips  (1997)  suggests  that  ROI  is  a  fifth- level  addition  to  Kirkpatrick’s  four- 
level  evaluation  model  (1998).  He  further  suggests  that  instructional  designers  who  have 
conducted  program  evaluation  using  Kirkpatrick’s  model  may  notice  strong  parallels 
between  this  expanded  view  of  ROI  analysis  and  each  level  of  Kirkpatrick’s  evaluation 
model.  Indeed,  the  design  of  the  front-end  ROI  analysis  and  the  program  evaluation  can 
and  should  be  conceptualized  simultaneously  so  that  inputs  and  outputs  to  the  assessment 
and  evaluation  efforts  are  consistent  (Phillips,  1997).  In  all  cases,  special  care  must  be 
given  to  ensure  that  changes  or  values  that  are  being  attributed  to  the  performance 
improvement  environment  are  not,  in  fact,  the  result  of  some  other  influence 
(Lachenmaier  &  Moor,  1997). 

This  chapter  reviewed  the  CBA  process  with  a  focus  on  the  basic  steps  of  a 
thorough  analysis.  Discussion  centered  on  concerns  associated  with  estimating  military 
value  and  the  specific  application  of  a  CEA  to  evaluating  military  versus  public  sector 
programs.  This  was  followed  by  a  review  of  intangible,  value-added  activities  that  are 
difficult  to  quantify,  but  nonetheless  important  to  any  analysis  of  training  and  education 
programs.  A  presentation  of  the  analysis  strategy  outlined  the  data  collected  from 
MCCES  and  provided  a  foundation  for  the  ROI  model.  The  ROI  process  model 
attempted  to  demonstrate  the  potential  benefits  of  incorporating  DL  technology  into  the 
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selected  courses  based  on  commonly  used  BCR  and  ROI  formulas.  These  calculations 
were  followed  by  a  discussion  of  some  additional  significant,  yet  hard  to  quantify 
variables  that  should  be  considered  as  part  of  the  total  analysis.  Before  making  a  final 
selection  of  instructional  media,  therefore,  it  is  important  to  determine  if  the  potential 
benefits  accrued  by  DL  or  technology  insertion  outweigh  the  potential  costs,  or  whether 
the  instruction  will  be  cost-effective  and  provide  an  acceptable  Return  On  Investment. 
The  purpose  of  a  cost  analysis  is  to  provide  the  necessary  information  to  facilitate  and 
improve  the  evaluation  and  decision-making  processes.  It  is  important,  therefore,  that  an 
accurate  and  comprehensive  analysis  of  cosCbenefit  data  be  accomplished. 
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VI.  FINDINGS 


A.  FINDINGS  FROM  T&E  DIVISION,  DL  BRANCH 


1.  Funding  Support 


Establishing  a  viable  DL  program  for  the  USMC  is  a  resource-intensive  project 
that  has  required  a  significant  investment  in  order  to  build  the  necessary  infrastructure 
enabling  access  for  all  Marines  and  Marine  Corps  civilian  personnel.  The  Marine  Corps 
DL  program  has  already  received  initial  program  funding  starting  in  FY99  to  establish  a 
representative  architecture  and  complete  the  pilot  initiative  (HQMC  T&E  DL 
ROADMAP,  1999).  Moreover,  the  DL  program  is  also  fully  funded  for  the  FY01-05 
time  frame  (HQMC  T&E  POM-OO). 

Although  the  funding  issue,  which  normally  represents  the  biggest  hurdle  to  a 
program’s  implementation  has  been  successfully  cleared,  the  Marine  Corps  DL  program 
is  far  from  fully  functional.  A  number  of  concerns  still  remain  that  must  be  addressed  if 
the  true  potential  of  DL  is  to  be  fully  recognized  by  the  Marine  Corps.  To  help  reduce 
the  overall  cost  of  the  program,  several  on-going  initiatives  and  programs  are  being 
leveraged  including  (HQMC  T&E  DL  ROADMAP,  1999): 

•  Base  Telecommunications  Infrastructure  (BTI)  -  This  initiative  upgrades  the 
telecommunications  network  infrastructure  aboard  every  base  and  station 
between  FY97  and  FY03.  The  BTI  is  already  fully  funded. 

•  Reserve  Information  Network  -  The  R-Net  provides  telecommunications 
connectivity  to  each  Marine  Corps  Reserve  site.  Total  R-Net  installation  has 
been  completed. 

•  Marine  Corps  Satellite  Education  Network  -  MCSEN  is  an  existing  video 
conferencing  network  available  aboard  major  Marine  Corps  bases  and 
stations.  MCSEN  is  currently  devoted  to  voluntary  off-duty  education  and  the 
academic  skills  program.  MCSEN  is  fully  funded. 

•  Total  Army  Distance  Learning  Program  -  The  TADLP  will  provide  the 
Marine  Corps  with  additional  DL  courseware  and  training  opportunities, 
including  access  through  the  Army’s  Doctrine  and  Training  Digital  Library 
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(ADTDL),  the  Training  Network  (TNET),  and  Satellite  Education  Network 
(SEN).  The  TADLP  is  funded. 

•  National  Guard  Bureau’s  Distributed  Training  Technology  Program  (DTTP)  - 
The  DTTP  establishes  a  robust  DL  infrastructure  across  all  50  states  and  is  a 
funded  initiative.  The  Marine  Corps  and  NGB  are  currently  exploring 
collaboration  opportunities  for  the  future. 

•  Navy  DL  Programs  -  CNET  is  providing  assistance  with  IMI  development 
and  connectivity  through  the  CNET’s  CESN.  The  Navy  DL  programs  are 
fully  funded. 

Taken  collectively,  each  of  these  initiatives  provide  a  broad  foundation  of 
integrated  networks  and  courseware  upon  which  the  DL  program  can  interact  and  expand 
as  needed  to  reach  target  audiences.  Although  not  specifically  addressed  in  this  study,  the 
cost  savings  recognized  from  the  exploitation  of  these  systems  alone  would  prove  quite 
substantial  since  the  hardware  and  supporting  infrastructure  are  already  in  place. 
Expanding  upon  the  potential  of  these  established  systems  rather  than  starting  from 
scratch  simply  makes  good  business  sense.  Based  on  my  study  of  the  training 
environment  at  MCCES,  however,  I  was  unable  to  further  access  the  potential  value  of 
incorporating  these  systems  into  the  USMC  DL  effort  as  the  DL  program  is  still  in  the 
pilot  stage  and  no  further  work  has  been  done  at  this  time  regarding  these  issues. 


2.  Funding  Assumptions 


In  addition  to  the  potential  savings  offered  by  the  use  of  the  above  programs, 
justification  for  DL  program  funding  was  based  upon  four  basic  assumptions  (HQMC 
T&E  DL  ROADMAP,  1999): 

1)  The  TEMI,  of  which  DL  is  a  part,  would  successfully  reduce  institutional 
training  time  by  approximately  30  percent. 

2)  Known  as  training  buy-back,  this  training  time  strategy  will  reduce  traditional 
resident  training  tracks  thereby  reducing  the  T2P2  account.  Marines  who 
finish  training  sooner  can  then  be  sent  to  the  operating  forces  faster,  thus 
improving  operational  readiness. 

3)  As  a  result  of  shorter  resident  training  tracks,  the  frequency  of  instruction  and 
throughput  numbers  can  be  increased  accordingly,  thus  reducing  the  number 
of  Marines  awaiting  training. 
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4)  Increased  use  of  DL  will  reduce  TAD  expenditures  associated  with  formal 
training  by  approximately  30  percent. 

B.  FINDINGS  FROM  MCCES 

Although  by  no  means  totally  inclusive,  these  results  represent  major  issues  of 
concern  to  the  instructors  and  administrators  at  MCCES  and  MCAGCC.  Specific 
findings  include: 

1.  Reductions  in  observed  course  lengths,  mandated  by  HQMC  and  chiefly 
attributable  to  CBT,  have  already  taken  place  at  MCCES.  These  reductions 
are  far  in  excess  of  the  industry  standard  of  30  percent  normally  associated 
with  DL  technology.  These  reductions  were  viewed  by  MCCES  personnel  as 
excessive,  arbitrary,  and  not  in  accordance  with  the  provisions  of  the  SAT 
and  TEMI. 

2.  Although  observed  course  lengths  have  been  systematically  reduced  and 
Marines  are  getting  to  operational  units  faster,  the  quality  of  those  Marines’ 
training  has  come  into  serious  question  by  both  the  schoolhouse  and  the  FMF. 

3.  Senior  instructors  tolerated  CBT,  but  felt  it  was  not  a  suitable  replacement  for 
conventional  training  methods.  Their  view  was  that  the  overall  quality  of 
instruction  was  hampered  by  reductions  in  conventional  student-instructor 
interaction  methods.  Junior  instructors  favorably  endorsed  CBT  as  the  best 
solution  to  the  problem  of  reduced  instructional  hours.  All  personnel 
interviewed  in  both  groups  had  been  at  MCCES  prior  to  the  activation  of  the 
AEC.  Both  groups  were  in  agreement  that  the  imposed  reductions  in  the 
curricula  were  excessive  and  sacrificed  critical  skills  necessary  to  maintain 
quality  instruction. 

4.  Although  no  DL  applications  have  been  implemented  beyond  the  pilot  stage, 
the  complete  hardware/software  infrastructure  (minus  content,  which  will  be 
discussed  below)  is  in  place  according  to  plan. 

5.  The  LAB  VOLT  experiment,  an  automated  self-paced  instructional  system  in 
electronic  theory,  considerably  reduced  the  overall  student 
recycle/remediation  rate  from  46  percent  to  6  percent.  Under  the  previous 
traditional  lecture  method,  approximately  80  percent  of  the  students  were  set 
back  at  least  once  during  the  course  of  instruction.  Although  originally 
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designed  as  a  stand-alone  system,  LAB  VOLT  has  been  found  to  work  most 
effectively  when  proceeded  by  a  traditional  lecture  period.  Based  on 
empirical  evidence,  it  was  found  that  no  additional  training  time  was  needed 
to  incorporate  lecture  back  into  the  overall  curricula,  as  the  students 
progressed  through  the  self-paced  lesson  at  an  increased  rate,  and  with  greater 
comprehension  when  the  instruction  began  with  a  short  lecture  period.  While 
not  directly  related  to  the  DL  program,  these  impressive  results  definitely 
have  technology-related  implications  such  as  increased  comprehension  and 
completion  rates. 

6.  It  was  the  unanimous  opinion  of  all  personnel  interviewed  at  MCCES,  that 
with  the  possible  exception  of  MAT,  DL  technology  had  no  application  to 
entry-level  skill  training.  This  was  attributed  to  the  fact  that  Marines  at  this 
level  of  training  are  most  receptive  to  and  in  need  of  the  personal  interaction 
techniques  of  conventional  classroom  instruction.  Historical  evidence 
suggests  that  Marines,  at  this  point  in  their  enlistment,  are  often  overwhelmed 
by  the  dramatic  change  in  their  environment  and  life  style.  Attendance  at 
their  formal  MOS  school  represents  the  first  real  independence  new  Marines 
are  given  following  the  rigors  of  basic  training  and  MCT.  To  expect  these 
young  men  and  women  to  sacrifice  the  little  free  time  they  are  given  in 
pursuit  of  additional  academic  challenges  is  somewhat  unrealistic. 
Furthermore,  any  type  of  non-MOS  instruction  outside  of  formal  training  was 
viewed  as  a  distraction  from  the  Marine’s  primary  duty  of  excelling  in  his 
class.  In  support  of  the  Marine’s  transformation  process,  and  in  the  interest  of 
imbuing  the  ideals  of  teamwork  and  unit  cohesion,  formal  schoolhouse 
training  is  therefore  considered  a  must  for  entry-level  training. 

7.  Company  A  introduced  a  maximum  training  effort  to  eliminate  its  MAT 
problem  and  realized  limited  success.  Their  solution  involved  increasing  the 
training  tempo  to  24  hours  per  day.  Problems  quickly  surfaced  that  included 
such  issues  as  equipment  failure  from  overuse,  and  instructor  fatigue.  It  was 
discovered  that  the  problem  was  never  really  solved,  only  pushed  further 
down  the  training  pipeline  to  succeeding  phases  of  training  that  could  not 
handle  the  increased  throughput.  Through  a  process  of  trial  and  error  the 
training  tempo  was  eventually  reduced  to  current  levels,  which  are  considered 
optimal. 
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8.  Although  significantly  reduced,  the  MAT  problem  still  existed  for  Company 
A.  In  response,  Company  A  trains  their  personnel  in  a  MAT  status  in  other 
non-MOS  essential,  yet  time  consuming-functions  such  as  drill  and 
ceremonies,  uniform  inspections,  physical  fitness,  etc.  Although  considered 
an  efficient  solution  to  the  MAT  problem,  this  approach  has  not  yet  been 
standardized  among  the  other  training  companies  at  MCCES. 

9.  Personnel  interviewed  at  MCCES  generally  agreed  that  individuals  in  a  MAT 
status  would  be  good  candidates  for  DL  technology  applications.  The  time 
would  be  advantageous  for  introductory  lessons  in  support  of  the 
transformation  process  such  as  personal  finance,  map  reading,  and  other 
general  knowledge  courses.  There  would  be  no  distraction  from  formal 
training  since  it  had  not  yet  begun,  so  the  Marines  could  devote  their  full 
attention  to  the  instruction.  Once  formal  MOS  related  training  began, 
however,  it  was  strongly  emphasized  that  all  DL  instruction  should  stop  so 
the  Marines  could  devote  their  full  time  and  attention  toward  that  effort. 

10.  Advanced-skill  level  courses  were  deemed  suitable  for  DL  applications. 
Distance  Learning  could  aid  in  refreshing  the  basic  skills  of  senior  Marines 
prior  to  formal  school  attendance.  This  action  would  not  only  allow  the 
Marines  to  begin  the  course  of  instruction  at  a  higher  skill  level,  but  would 
contribute  to  faster  progression  and  comprehension  rates,  leading  to  possible 
course  reduction.  Simply  stated,  preassessment  and  course  compression 
would  be  enhanced. 

11.  Instructors  at  MCCES  felt  that  CBT  and  DL  technologies  should  not  be  used 
as  a  blanket  excuse  to  automatically  reduce  the  length  of  courses.  The 
Marine  Corps  is  continually  fielding  newer  and  more  complex  equipment  as 
compliments  to,  rather  than  replacements  for,  existing  equipment.  As  such, 
Marines  are  expected  to  be  intimately  familiar  with  complex  networks  of 
technologically  sophisticated  equipment.  Where  and  when  are  Marines  to  be 
taught  how  to  operate  and  maintain  this  increasing  inventory  of  equipment? 
MCCES  personnel  suggest  that  training  timesavings  realized  from  CBT  and 
DL  technologies  should  not  be  automatically  eliminated  from  curricula,  but 
should  be  maintained  for  inclusion  of  additional  necessary  information. 

12.  MCCES  uses  inaccurate  costing  data.  The  cost  of  an  individual  Marine 
training  day,  estimated  at  $271,  had  no  supporting  historical  proof.  No  one  at 
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MCCES  could  explain  where  this  value  came  from  or  under  what 
assumptions  it  had  originally  been  calculated. 

13.  Based  on  my  assumptions,  I  calculated  the  cost  of  a  Marine  training  day  to  be 
approximately  $82.00.  Combining  this  value  with  previously  calculated  data 
on  the  cost  of  training  first-term  Marines,  I  was  able  to  calculate  a  BCR  and 
ROI  for  three  observed  courses  at  MCCES.  Results  were  favorable  with  a 
positive  ROI  that  averaged  approximately  147  percent  for  the  three  selected 
courses,  or  an  average  of  $1.47  returned  for  every  dollar  invested.  Elements 
of  a  CEA  were  also  included  in  my  analysis,  which  indicated  potential 
favorable  results  for  DL  applications  in  terms  of  the  military  value  added. 
Distance  Learning  technology  has  the  potential  to  keep  senior  Marines  in 
operational  units  for  a  longer  period  of  time,  while  affording  them  the  same 
opportunities  for  learning  that  school  trained  individuals  currently  have. 
Distance  Learning  applications  can  thus  be  viewed  as  a  value  added  activity 
that  enhances  the  cost  effectiveness  of  a  training  program  and  promotes  unit 
readiness. 

14.  SMEs  chosen  for  CCRBs  are  currently  chosen  on  the  basis  of  seniority,  and 
are  not  necessarily  the  most  experienced  individuals  in  their  field.  In  light  of 
the  rapid  rate  at  which  modem  technology  continues  to  advance,  many 
systems  are  fielded  that  the  most  senior  personnel  in  a  unit  know  little  or 
nothing  about.  These  same  individuals,  however,  are  the  ones  who  are 
chosen,  as  SMEs,  to  make  decisions  concerning  which  critical  skills 
(regarding  these  new  systems)  will  be  taught  at  the  schoolhouse  and  which 
will  be  handled  in  the  FMF.  They  obviously  push  for  the  schoolhouse  to 
conduct  the  majority  of  the  instruction.  As  expected,  a  dilemma  ensues  when 
the  responsibility  is  placed  on  the  schoolhouse  to  teach  an  increasing  number 
of  these  critical  skills,  while  simultaneously  the  length  of  the  curricula  is 
disproportionately  reduced. 

15.  A  review  of  MOE  regarding  CBT  (i.e.,  test  scores),  from  three  random 
classes  prior  to  the  activation  of  the  AEC  and  three  random  classes  after, 
found  no  significant  difference  in  the  performance  of  the  students.  These 
findings  applied  to  both  entry-level  and  advanced-level  training  courses 
observed  at  MCCES. 
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16.  A  fully  operational  VTT  facility  was  located  at  MCAGCC,  but  no  plans  had 
been  designed  to  exploit  its  potential  as  part  of  a  DL  program. 

17.  The  LRC,  although  folly  operational  at  both  MCCES  and  MCAGCC,  was 
significantly  underused  due  to  a  lack  of  content.  The  number  of  automated 
courses  currently  available  totaled  only  three,  and  failed  to  generate  sufficient 
interest  due  to  software  compatibility  problems  as  well  as  difficulties  with 
electronic  testing  procedures.  MCI,  which  has  total  responsibility  for  course 
conversion  procedures,  has  yet  to  move  beyond  the  initial  pilot  stage  of  the 
program. 

18.  Insufficient  historical  data  regarding  the  USMC  DL  program  currently  exist 
to  determine  the  actual  contribution  DL  can  make  to  the  Marine  Corps.  This 
is  due  to  the  fact  that  the  program  has  not  yet  proceeded  beyond  the  initial 
pilot  stage,  although,  as  mentioned,  the  supporting  infrastructure  is  in  place 
and  ready  for  use. 
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VII.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

1.  DL  Program  Funding  Assumptions 

The  previously  discussed  funding  assumptions,  upon  which  DL  program  was 
justified,  represent  the  logic  behind  the  entire  process.  I  will  therefore  focus  my 
discussion  around  these  four  main  points. 

a.  Training  Time  Reduction 

The  industry  standard  of  a  30  percent  reduction  in  conventional  training 
time,  through  the  use  of  DL  technology,  represents  a  sound  benchmark  for  a  cost 
analysis,  but  this  figure  is  by  no  means  always  dependable.  It  was  found  that  all  of  the 
courses  reviewed  at  MCCES  had  already  been  reduced  by  more  than  30  percent  due  to 
the  introduction  of  CBT,  as  well  as  improvements  in  instructional  efficiency  (Taylor, 
2000).  Some  of  these  reductions  were  viewed  as  capricious  and  excessive,  impacting  on 
the  overall  quality  of  the  course  and  the  ability/skill  level  of  the  graduates.  Although  my 
own  analysis  was  based  on  a  similar  standard  of  30  percent  and  produced  favorable 
results,  the  issue  remains  questionable  until  DL  technology  is  actually  implemented  and 
hard  data,  in  terms  of  DL  course  enrollment  and  completion  rates,  become  available. 

In  some  cases,  further  reductions  in  course  length  of  any  kind  could 
eliminate  crucial  skills  that  cannot  be  successfully  learned  other  than  through  face-to-face 
interaction.  Evidence  has  been  presented  of  deficiencies  in  the  skill  levels  of  new 
Marines  leaving  MCCES  that  must  be  taught  in  the  operational  units.  Many  of  these 
difficulties  have  been  traced  to  miscommunication  between  SMEs  at  the  schoolhouse  and 
in  the  FMF  over  critical  ITS,  and  responsibility  for  teaching  them.  More  importantly,  the 
spirit  of  the  SAT  and  TEMI,  which  comprise  the  foundation  of  curricula  development 
procedures  for  the  Marine  Corps  (and  were  not  followed  at  MCCES),  should  be  more 
closely  adhered  to  when  converting  course  content  to  a  DL  format.  This  should  help 
ensure  that  similar  mistakes  are  not  repeated  with  the  DL  program,  and  critical  skills  most 
suitable  for  a  formal  classroom  environment  are  not  eliminated  in  favor  of  faster 
completion  rates. 
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Once  sufficient  content  is  available  for  practical  use  of  DL  technology,  the 
Marine  Corps  needs  to  promote  this  form  of  training  and  education  with  the  utmost 
enthusiasm.  Failure  to  gain  the  support  of  senior  personnel,  both  within  the  formal 
school  system  as  well  as  the  FMF,  could  seriously  affect  buy-in  of  the  DL  program 
among  the  rank  and  file.  Linking  the  use  of  DL  programs  with  increased  advancement 
opportunities  is  but  a  first  step  toward  promoting  the  acceptance  and  use  of  this  training 
process. 

In  a  similar  vein,  Marines  no  longer  have  the  luxury  to  wait  until  afforded 
the  opportunity  to  attend  a  formal  school  and  expect  the  faculty  to  provide  all  the 
necessary  information  for  proficiency  in  a  relatively  limited  period  of  time.  Familiarity 
with  elementary  computer  skills  and  basic  MOS  knowledge  must  already  be  current  prior 
to  attending  school.  The  burdensome  operational  tempo  most  Marines  face  in  today’s 
environment  certainly  does  not  provide  the  individual  with  the  time  necessary  to  acquire 
many  of  these  skills,  prior  to  attending  formal  training,  yet  this  situation  by  no  means 
relieves  the  individual  of  the  responsibility  for  knowing  them.  This  circumstance  is  not 
much  different  than  that  faced  by  civilian  counterparts  who  are  continually  required  to 
refresh  their  skills  to  stay  competitive,  but  are  reluctant  to  take  time  from  their  jobs  to  do 
so.  The  individuals  who  successfully  accomplish  this  formidable  task  are  the  ones  who 
can  most  efficiently  manage  their  time.  Distance  Learning  technologies  can  significantly 
affect  one’s  ability  to  manage  time  more  efficiently  for  training  purposes. 

Distance  Learning  programs  can  also  greatly  assist  in  eliminating  the 
unfamiliarity  problem  that  many  senior  Marines  experience  with  fielding  new  equipment 
in  the  FMF.  Those  senior  Marines  who  are  expected  to  be  the  most  knowledgeable  in 
regard  to  new  systems,  and  yet  have  the  least  opportunity  to  attend  formal  school,  can  use 
DL  technology  to  maintain  their  MOS  skills.  A  value  added  consideration  here  is  that 
DL  technology  can  provide  badly  needed  expertise  to  senior  Marines,  who  may  find 
themselves  chosen  as  SMEs,  and  must  recommend  which  skills  to  teach  at  school  and 
which  to  handle  in  the  FMF.  Familiarity  with  DL  programs  will  thus  allow  SMEs  to 
make  better-informed  decisions  regarding  training  and  education  programs. 

b.  Training  Buy-back 

The  issue  of  training  buy-back  and  its  effect  on  the  T2P2  account  is  also 
somewhat  questionable.  The  previous  CMC  specifically  directed  MCCES  and  all  other 
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formal  Marine  Corps  schools  to  reduce  the  length  of  courses  to  get  Marines  to  the 
operating  forces  faster  and  thus  improve  operational  readiness.  The  first  part  of  this 
guidance  was  readily  achieved,  but  the  impact  on  operational  readiness  remains  in  doubt 
for  reasons  previously  discussed.  Marines  are,  in  fact,  getting  to  operational  units  faster, 
but  the  quality  of  their  training  has  come  into  serious  question.  Based  on  my  research  at 
MCCES,  this  problem  was  again  traced  to  an  improper  needs  assessment  by  SMEs,  who 
failed  to  follow  the  procedures  of  the  SAT  and  TEMI.  Whether  DL  initiatives  will 
positively  affect  training  buy-back  depends  upon  a  thorough  analysis  by  qualified  SMEs 
of  those  critical  skills  that  must  remain  in  the  schoolhouse,  and  those  that  may  be  applied 
in  a  DL  environment. 

Short  of  total  mobilization  for  war,  the  Marine  Corps  continually  operates 
with  a  Table  of  Organization  (T/O)  well  below  its  authorized  manning  level.  Although 
they  may  not  like  it,  unit  commanders  have  become  accustomed  to  this  situation  and  in 
most  circumstances  manage  to  meet  their  obligations  with  the  personnel  and  equipment 
that  they  have.  As  such,  the  FMF  does  not  miss  the  Marines  it  has  not  yet  received  from 
the  training  pipeline.  Efforts  to  move  entry-level  Marines  through  the  training  cycle  more 
quickly  is  a  laudable  goal,  but  only  if  the  quality  of  the  training  those  Marines  receive  is 
not  sacrificed  for  the  sake  of  an  increased  rate  of  throughput.  It  does  no  good  to  get  a 
Marine  to  the  FMF  more  quickly  if  that  Marine  can’t  satisfactorily  perform  his  job. 
Furthermore,  unit  readiness  suffers  as  training  priorities  must  now  be  changed  to  teach 
the  new  Marine  the  skills  he  should  have  learned  in  school.  Evidence  gathered  from 
MCCES  indicates  that  this  has  been  the  case  in  some  instances. 

The  success  of  the  LAB  VOLT  experiment,  although  not  directly  related  to 
DL  technology,  demonstrates  the  potential  of  automation  to  increase  the  efficiency  of 
training  and  education  programs.  Distance  Learning  for  the  Marine  Coips  is  by  no  means 
viewed  as  a  panacea  for  the  ailments  of  traditional  classroom  instruction,  but  rather  as  a 
complement  that  will  serve  to  benefit  all  Marines.  Just  as  the  LAB  VOLT  system  was 
found  to  function  most  efficiently  when  combined  with  limited,  traditional  lecture 
periods,  DL  will  provide  the  greatest  benefit  when  properly  combined  with  current 
training  methods. 

In  a  rush  to  apply  the  timesaving  benefits  of  CBT  and  DL  technologies, 
the  Marines  Corps  needs  to  consider  the  warnings  of  the  personnel  tasked  with  working 
with  these  technologies.  With  the  increased  amount  of  information  that  all  Marines  are 
responsible  for  in  today’s  technologically  advanced  environment,  formal  schools  find 
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themselves  taxed  to  teach  more  in  less  time.  Automated  technologies  can  certainly  assist 
in  this  effort,  but  the  advantages  gained  through  the  technology  can  quickly  disappear  if 
there  is  insufficient  time  to  teach  vital  information.  Compromises  will  have  to  be  made 
on  both  sides  of  the  issue,  but  the  Marine  Corps  needs  to  ensure  that  training  and 
education  programs  are  optimized  to  meet  the  needs  of  the  individual  Marine  and  are  not 
solely  focused  on  increased  throughput. 

c.  Marines  Awaiting  Training 

Based  on  initiatives  undertaken  by  instructors  at  MCCES,  the  issue  of 
MAT  has  been  addressed  with  mixed  results.  Placing  Marines  in  a  training  hiatus  once 
their  course  had  begun  was  found  to  have  adverse  effects  on  the  retention  of  knowledge 
learned  from  earlier  phases  of  training  (Herring,  2000).  It  was  therefore  determined  that  it 
was  in  the  best  interests  of  the  students  to  place  them  in  a  MAT  status  prior  to  the  start  of 
a  course  rather  than  interrupt  the  training  cycle  once  the  POI  had  begun.  The  contribution 
that  DL  can  make  to  this  situation  is  perhaps  the  only  positive  area  in  regard  to  entry- 
level  training.  It  was  the  unanimous  opinion  of  all  personnel  interviewed  at  MCCES  that 
once  formal  training  began  for  entry-level  Marines,  DL  technology  had  little  application 
due  to  the  need  to  maintain  academic  discipline  and  enhance  knowledge  transfer  among 
new  Marines. 


d.  TAD  Reduction 

The  assumption  that  increased  use  of  DL  will  reduce  TAD  expenditures 
associated  with  formal  training  by  30  percent  is  based  on  industry  standard,  and  will 
remain  as  such  until  hard  data  becomes  available  for  the  Marine  Corps  to  either  support 
or  refute  this  claim.  A  review  of  over  33  empirical  studies  of  civilian  businesses  involved 
with  DL  over  the  past  decade  has  found  this  figure  to  be  quite  reliable  (Johnston  & 
Fletcher,  1997).  Although  easily  quantifiable  cost  figures  such  as  these  are  most 
frequently  promoted  as  justification  for  DL  technology,  this  paper  does  not  suggest  that 
the  Marine  Corps  should  continue  with  its  implementation  plan  simply  because  the 
figures  and  the  technology  have  pushed  it  there.  Barring  any  insurmountable  obstacles  to 
full  implementation,  however,  which  this  study  has  not  found,  the  current  plan  for  the  DL 
program  should  be  continued. 
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2.  Summary 


Based  on  my  observations  at  MCCES,  significant  progress  has  been  achieved  in 
increasing  instructional  efficiency  through  the  use  of  automation  in  the  classroom.  When 
properly  applied,  DL  technology  will  continue  this  trend  with  substantial  positive  returns 
on  the  investment.  A  number  of  hurdles  will  have  to  be  cleared  before  any  positive  ROI 
will  be  realized.  For  the  Marine  Corps,  DL  technology  is  not  applicable  to  entry-level 
training,  and  with  the  possible  exception  of  personnel  in  a  MAT  status,  should  no  longer 
be  considered  for  this  category  of  training.  Distance  Learning  technology  does  have 
enormous  potential  in  the  Marine  Corps,  however,  for  advanced  skill  level  training  and 
should  be  implemented  to  the  fullest  extent  possible  as  soon  as  circumstances  permit. 
The  potential  savings  that  DL  can  offer  through  preassessment  training  and  course 
compression  have  been  briefly  considered  in  this  paper,  and  definitely  merit  further  study 
as  more  data  become  available.  This  implementation  procedure  should  not  be  too  hasty, 
as  a  thorough  needs  assessment,  in  accordance  with  the  SAT  and  TEMI,  is  critical  to 
identify  the  crucial  skills  that  can  be  taught  most  effectively  in  a  formal  schoolhouse 
environment. 

B.  RECOMMENDATIONS 

INCREASE  MCI  INVOLVEMENT.  It  was  observed  that  the  LRCs  at  MCAGCC 
were  currently  underutilized  for  lack  of  adequate  electronic  courses  of  instruction.  The 
number  of  courses  available  totaled  only  three,  and  failed  to  generate  sufficient  interest 
due  to  software  compatibility  problems  as  well  as  difficulties  with  electronic  testing 
procedures.  I  am  confident  that  once  the  MCI,  which  has  sole  responsibility  for  this  area, 
works  out  these  software  problems  and  fields  more  courses,  the  full  potential  of  this  DL 
entity  will  be  fully  realized.  Further  research  into  a  cost  analysis  of  the  conversion 
program  that  MCI  is  currently  using,  is  definitely  warranted. 

EXPAND  YTT  USE.  It  was  also  discovered  that  although  MCAGCC  has  a  fully 
functional  VTT  site,  nothing  is  currently  being  done  to  use  this  technology  for  DL 
applications.  The  benefit  of  such  technology  has  been  well  documented  by  Cabrera 
(1999)  and  rates  further  study. 
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FOCUS  ON  THE  RESERVE  COMPONENT.  Additional  research  is  particularly 
needed  in  regard  to  DL  applications  and  the  reserve  component  of  the  Marine  Corps. 
Discussions  with  schoolhouse  personnel  at  MCCES  revealed  several  areas  of  interest 
where  DL  was  currently  being  studied  for  use  in  training  reserve  Marines.  Marine  Forces 
Reserve  (MARFORRES)  has  recently  invested  considerable  funding  into  a  prototype  of 
the  LAB  VOLT  system  for  use  at  selected  reserve  centers  throughout  the  country 
(Herring,  2000).  Distance  Learning  applications  were  found  to  be  particularly  applicable 
to  advanced  skill  level  training  where  reservists  could  become  proficient  on  the  basics  of 
their  MOS  while  at  their  parent  units.  Such  application  would  thus  permit  more  efficient 
use  of  the  two-week  active  duty  period  in  the  summer  months,  when  most  reservists 
usually  attended  formal  schooling  (Foy,  2000).  A  study  was  conducted  by  the  Marine 
Corps  regarding  the  potential  of  DL  technology  applications  for  the  reserve  component 
(Metzko,  et.al.,  1996),  yet  based  on  my  research  little  has  been  accomplished  since.  It  is 
definitely  time  for  a  more  current  study  to  see  where  DL  can  most  significantly  contribute 
to  the  reserve  component. 

EXPLORE  DL  APPLICATIONS  TO  MAT.  If  DL  is  to  be  used  at  all  for  entry- 
level  training,  the  most  appropriate  time  should  be  for  personnel  in  a  MAT  status.  The 
application  of  DL  technology  and  MAT  definitely  warrants  further  study  for  potential 
training  dividends. 

ENCOURAGE  PERSONAL  INITIATIVE.  The  Marine  Corps  must  continue  to 
emphasize  the  importance  and  necessity  for  the  individual  Marine  to  focus  on  his  own 
educational  advancement.  As  a  result  of  increased  training  requirements  combined  with 
reductions  in  formal  school  courses,  it  is  incumbent  upon  the  individual  Marine  to  pursue 
education  and  training  on  his  own  initiative  more  than  ever  before.  Distance  Learning 
applications  can  be  of  use  in  this  effort. 

UTILIZE  TTMESAVINGS  EFFICIENTLY.  Refrain  from  viewing  training 
technology  solely  as  a  curricula-cutting  measure.  The  timesavings  realized  from  the  use 
of  CBT  and  DL  technology  should  not  be  automatically  eliminated  from  established 
training  schedules.  Such  technology  provides  the  greatest  benefit  when  used  to 
complement  rather  than  replace  conventional  training  methods. 

ELIMINATE  INSTITUTIONAL  BIAS.  To  fully  realize  the  potential  of  DL,  the 
Marine  Corps  must  deal  with  several  challenges.  First,  the  anticipated  resistance  to 
change  embedded  within  the  institution,  although  found  to  be  minimal,  must  be 
overcome.  Distance  Learning  establishes  a  new  learning  environment  and  considerably 
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changes  established  training  and  education  paradigms.  Such  changes  will  naturally  meet 
resistance  due  to  lack  of  experience  with  curriculum  development  procedures  developing 
effective  DL  solutions,  unfamilianty  with  the  technology,  and  the  erroneous  perception 
that  DL  will  eliminate  jobs  or  diminish  resident  instruction.  Lack  of  personal  experience 
appears  to  be  the  key  reservation  among  detractors  of  DL  programs.  The  Marine  Corps 
must  aggressively  endorse  the  value  and  use  of  DL  technology. 

PROMOTE  PERSONAL  INCENTIVES.  To  promote  effective  buy-in  of  DL 
technology  among  Marines,  a  system  of  incentives  for  DL  programs  that  is  tied  to 
promotion  or  advancement  opportunities  must  be  established.  Such  incentives  will  most 
likely  provide  the  catalyst  for  rapid  expansion  of  current  DL  initiatives. 

PROVIDE  GUIDANCE  AND  MENTORSHIP.  Providing  effective  mentoring 
and  maintaining  cohesion  in  a  distributed  training  environment  will  prove  challenging 
The  shift  to  student-centered  learning  must  be  balanced  with  a  team  training  approach 
that  is  supported  and  supervised  by  senior  leaders.  This  shift  in  learning  methods  will  by 
no  means  occur  overnight,  but  will  require  several  years  to  implement.  Furthermore,  not 
every  Marine  will  learn  at  the  same  pace  using  this  approach.  Some  Marines  will  require 
a  more  structured  learning  environment  along  with  increased  personal  interaction  to 
facilitate  learning.  Any  training  and  education  solution  that  is  developed  must  be  flexible 
enough  to  accommodate  alternate  learning  methods  and  schedules.  The  key  ingredient  to 
this  concept  will  be  establishing  an  environment  at  the  unit  level  that  is  conducive  to 
learning  and  providing  the  proper  leadership  to  ensure  that  DL  programs  are  successfully 
completed. 

FOLLOW  THE  SAT  AND  TEMI.  Evidence  gathered  from  MCCES  reveals  that 
many  of  the  problems  encountered  when  applying  automated  training  technology  resulted 
from  a  failure  of  experienced  SMEs  to  properly  follow  established  procedures.  These 
publications  clearly  outline  the  proper  method  for  developing,  changing,  and  updating 
curricula.  Following  these  procedures  when  converting  existing  course  materials  to  a  DL 
format,  or  when  developing  new  curricula,  will  avoid  many  of  the  problems  encountered 
in  this  study.  Furthermore,  as  indicated  in  these  directives,  SMEs  should  be  chosen  based 
on  experience  levels,  rather  seniority  alone.  A  more  careful  screening  of  SMEs  will 
avoid  many  of  the  pitfalls  associated  with  an  improper  needs  assessment,  when  reviewing 
curricula  for  DL  conversion. 

FURNISH  ACCESS  TIME.  Issues  regarding  Internet  access  aboard  Marine 
Corps  bases  and  stations  must  also  be  addressed  and  resolved  prior  to  realizing  the  full 
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benefit  of  DL.  A  recent  study  by  the  Army  Research  Institute  (ARI,  1998)  revealed  that 
over  70  percent  of  the  officers  interviewed  had  regular  Internet  access,  while  only  30 
percent  of  the  enlisted  soldiers  interviewed  had  similar  Internet  access.  Accordingly,  the 
Army  has  identified  lack  of  Internet  access  for  soldiers  as  a  critical  barrier  to  success  in 
their  DL  program.  Although  the  Marine  Corps  has  not  yet  conducted  a  similar  study,  it 
seems  logical  that  similar  percentages  could  be  applied  to  the  Marine  Corps  personnel.  In 
order  for  DL  to  fully  succeed  therefore,  Marines  must  have  significantly  greater  access  to 
the  Marine  Corps  network  and  the  Internet. 

High  operational  tempo  also  makes  finding  additional  time  to  train  and  educate 
very  difficult.  Finding  time  for  Marines  and  civilian  personnel  to  learn  represents 
perhaps  the  greatest  challenge  to  leaders,  trainers  and  educators  in  the  information  age. 
DL  programs  cannot  be  relegated  to  off-duty  hours.  If  the  Marine  Corps  is  to  realize  the 
full  potential  of  new  instructional  delivery  methods,  deliberate  steps  must  be  taken  to 
schedule  training  and  education  events  into  normal  duty  hours. 

REVIEW  COHORT  QUALITY  &  DL.  A  final  consideration  of  my  study  regards 
the  quality  of  the  Marines  attending  MCCES.  The  quality  of  the  training  cohorts 
attending  MCCES  is  among  the  highest  in  the  Marine  Corps,  as  these  types  of  courses 
require  higher  Armed  Forces  Qualification  Test  (AFQT)  scores  to  successfully  meet  the 
demands  of  the  billets  (i.e.,  to  maintain  and  repair  complex  electronic  and  technical 
equipment).  As  such,  it  would  be  beneficial  to  see  if  a  relationship  exists  between  the 
success  rate  that  Marines  experience  with  DL  technology  and  their  level  of  intelligence. 
It  would  be  interesting  to  examine  the  success  rates  with  CBT  and  DL  applications  at 
other  formal  schools  within  the  Marine  Corps.  Also,  with  increasing  numbers  of  non- 
high  school  graduates  entering  the  Marine  Corps,  it  will  be  equally  interesting  to  see  how 
well  prepared  these  Marines  of  the  future  will  be  for  an  automated  training  environment, 
particularly  one  that  emphasizes  self  study  through  DL  technology.  Once  sufficient  data 
from  DL  applications  become  available,  further  research  should  be  conducted  in  this 
intriguing  area  of  training  and  education. 

BE  PATIENT.  Due  to  the  fact  that  the  Marine  Corps  has  yet  to  actively 
implement  any  DL  programs  beyond  the  initial  pilot  stage,  there  is  a  lack  of  sufficient 
data  to  do  a  more  thorough  analysis.  Although  the  early  results  of  automation  in  the 
classroom  have  been  positive  regarding  ROI  and  the  contribution  that  DL  technology  can 
make  appears  promising,  it  is  clearly  much  too  early  to  reach  any  final  conclusions.  The 
USMC’s  DL  roadmap  for  the  future  must  be  given  sufficient  time  to  be  fully 
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implemented  and  allowed  to  reach  fruition  before  its  ultimate  utility  can  be  properly 
judged.  The  issue  should  therefore  be  revisited  in  about  three  to  five  years. 
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APPENDIX  A.  ROI  DATA  FOR  USMC  MILITARY  OCCUPATIONAL 
SPECIALTIES  COMPILED  AT  HQMC,  T&E  DIVISION,  DL  BRANCH,  1999 
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5811  A10RF31  MILITARY  POLICE  (USMC)  1064 _ 63 _ 44 _ 67,032 _ 46,922  129 _ 55  $3,856,636  $1,652,844  5811 

3381  A1433L1  FOOD  SERVICE  SPECIALIST  1120 _ 58 _ M _ 64,960  45,472 _ 125 _ 53  $3,737,425  $1,601,753  3381 

6412  N23A952  AVIONICS  TECHNICIAN  I  589  105  74  61,845  43,292  119  51  $3,558,205  $1,524,945  6412 

LEVEL  CLASS  A1 


3043  I  M0330V1  IENLISTED  SUPPLY  BASIC  I  1020 
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APPENDIX  B.  ABBREVIATIONS  AND  ACRONYMS 


ACSDE 

American  Center  for  the  Study  of  Distance  Education 

ADTDL 

Army  Doctrine  and  Training  Digital  Library 

AEC 

Automated  Electronic  Classroom 

AFIT 

Air  Force  Institute  of  Technology 

AFQT 

Armed  Forces  Qualification  Test 

ALC 

Area  Learning  Center 

AOB 

Average  on  Board 

BFWC 

Basic  Field  Wireman’s  Course 

BTI 

Base  Telecommunications  Infrastructure 

BSNCOC 

Battle  Staff  Noncommissioned  Officer’s  Course 

CBA 

Cost  Benefit  Analysis 

CBT 

Computer  Based  Training 

CCRB 

Course  Curriculum  Review  Board 

CDD 

Course  Descriptive  Data 

CD-ROM 

CD-Read-Only  Memory 

CEA 

Cost  Effectiveness  Analysis 

CESN 

CNET  Electronic  Schoolhouse  Network 

CMC 

Computer-mediated  Communications 

CNA 

Center  for  Naval  Analyses 

CNET 

Chief  of  Naval  Education  and  Training 

CNO 

Chief  of  Naval  Operations 

CSCC 

Communications  Systems  Chief  Course 

DII 

Defense  Information  Infrastructure 

DISN 

Defense  Information  Systems  Network 

DL 

Distance  Learning 

DLC 

Distance  Learning  Center 

DoD 

Department  of  Defense 

DTTP 

Distributed  Training  Technology  Program 

EO 

Executive  Order 

EPSS 

Electronic  Performance  Support  Systems 

ET 

Embedded  Training 

FLC 

Functional  Learning  Center 

FMF 

Fleet  Marine  Force 

GFE 

Government  Furnished  Equipment 

GFL 

General  Facilitative  Links 

IDA 

Institute  for  Defense  Analyses 

IDEA 

Institute  for  Defense  Education  and  Analyses 

IMI 

Interactive  Multimedia  Instmction 

IT 

Information  Technology 

ITS 

Individual  Training  Standards 
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ISD 

Instructional  Systems  Development 

LCCE 

Life  Cycle  Cost  Estimate 

MARFORRES 

Marine  Forces  Reserve 

MAT 

Marines  Awaiting  Training 

MCAGCC 

Marine  Corps  Air  Ground  Combat  Center 

MCCES 

Marine  Corps  Communications  and  Electronics  School 

MCI 

Marine  Corps  Institute 

MCSC 

Marine  Corps  System  Command 

MCSEN 

Marine  Corps  Satellite  Education  Network 

MCT 

Marine  Combat  Training 

MCTFS 

Marine  Corps  Total  Force  System 

MOE 

Measure  of  Effectiveness 

MOS 

Military  Occupational  Specialty 

N/MCI 

Navy/Marine  Corps  Intranet 

NPV 

Net  Present  Value 

OCCC 

Operations  Communications  Chief  Course 

ODUSD(R) 

Office  of  the  Deputy  under  Secretary  of  Defense  (Readiness) 

OUSDP&R 

Office  of  the  Under  Secretary  of  Defense  for  Personnel  and 
Readiness 

PCS 

Permanent  Change  of  Station 

POI 

Program  of  Instruction 

POM 

Program  Objective  Memorandum 

R-NET 

Reserve  Information  Network 

ROI 

Return  on  Investment 

PME 

Professional  Military  Education 

R&D 

Research  and  Development 

RDC 

Regional  Development  Center 

SAT 

Systems  Approach  to  Training 

SEN 

Satellite  Education  Network 

SFL 

Specific  Facilitative  Links 

SME 

Subject-Matter  Expert 

SORTS 

Status  of  Resources  and  Training  Systems 

T2P2 

Transients,  Trainees,  Patients,  and  Prisoners 

T&E 

Training  and  Education 

TAD 

Temporary  Additional  Duty 

TADLP 

Total  Army  Distance  Learning  Plan 

TCP/IP 

Transmission  Control  Protocol/Intemet  Protocol 

TDY 

Temporary  Duty 

TEMI 

Training  and  Modernization  Initiative 

TEPOP 

Training  and  Education  Point  of  Presence 

TIP 

Training  Input  Plan 

TNET 

Training  Network 
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TRADOC 

TTC 

VR 

VTC 

VTT 


Training  and  Doctrine  Command 
Technician  Theory  Course 
Virtual  Reality 
Video  Teleconferencing 
Video  Teletraining 
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